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EXECUTIVE SUMMARY 


This report examines outcomes for grade- 12 students in academic years (AY) 2010—11, 
2011—12, and 2012—13 who were enrolled in a Linked Learning certified pathway (LLCP) 1 in 
California. Outcomes include student engagement in learning, measured by high school 
attendance and discipline events, as well as college readiness and postsecondary enrollment. 

The analyses in this report are conducted using observational data. Therefore, quasi- 
experimental statistical methods are used along with crosstabular analysis to compare 
outcomes for these students with outcomes for similar students who did not participate in a 
pathway program. 2 

Linked Learning pathways are offered in California Partnership Academies, National Academy 
Foundation academies, small theme-based high schools, and small learning communities within 
comprehensive high schools. Linked Learning pathway programs may elect to undergo 
certification by ConnectEd: The California Center for College and Career (ConnectEd) 3 and its 
partners, the College and Career Academy Support Network (CCASN), the National Career 
Academy Coalition (NCAC), the National Academy Foundation (NAF), and Education Trust- 
West. 4 The certification process involves the submission of documentation on program 
implementation, staff characteristics and practices, student selection, and program outcomes as 
well as onsite classroom observations and interviews with students and their parents, staff, and 
local business partners. 5 Outcomes for students attending an LLCP are the focus of this report. 

Data for the analyses are drawn from the Linked Learning District Initiative (LLDI), 6 which 
provides grants to nine California school districts for the development and implementation of 
Linked Learning pathways. The LLDI collects data about student academic and demographic 
characteristics and a variety of high school and postsecondary outcomes. These student-level 
data were supplemented by school-level information from the California Department of 


1 http://www.connectedcalifornia.org/ schools_districts/ certification 

2 The statistical approach is described in the section on Statistical Approach-Matching. 

3 http://www.connectedcalifornia.org/home 

4 The National Center for Innovation in Career and Technical Education (NCICTE) draws on the 
expertise of nationally recognized education researchers and practitioners from a range of organizations. 
ConnectEd is one of the team members included in the center partnership. To preserve objectivity in 
the conduct of this study, ConnectEd was excluded from the development and review of the study and 
its products. 

5 The certification criteria are shown in appendix H. 

6 http://www.connectedcalifornia.org/ schools_districts/ district_mitiative 
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Education (CDE) on school enrollment, the percentage of students eligible to participate in 
the National School Lunch Program (NSLP) (an indicator of family poverty), the percentage 
of English Language Learners (ELLs) enrolled, the percentage of racial and ethnic minority 
students, and teachers’ average number of years of experience. The federal Common Core of 
Data (CCD ) 7 provided information on the urbanicity of the school. 

Data for six of the nine LLDI districts 8 were used in the analyses conducted for this report. 
Because student participation in a Linked Learning pathway is voluntary, the characteristics 
of the students who participate are likely different from the characteristics of those who do 
not. Differences between these groups suggest that they are not equivalent, making 
comparisons between them inappropriate. Propensity score matching (PSM) is a quasi- 
experimental statistical method to correct for these differences by matching intervention 
participants to students who did not participate in the intervention, based on the probability 
of participation . 9 In the present study, propensity scores (i.e., the probability of participating 
in a Linked Learning pathway) were estimated for all students, and then used to match LLCP 
students with other students with similar propensities for participation. The result of this 
matching process is a group of LLCP students and a comparison group that are similar on 
observable characteristics, such as pre-high school academic achievement, measured by 
scores on grade-eight California Standards Tests (CSTs) in mathematics and English, 
demographic characteristics, such as race/ethnicity, disability status, gender, and 
participation in the federal NSLP, among other factors. 

Although PSM can produce groups that are similar to each other (balanced) on these 
characteristics, matching does not necessarily produce groups balanced on unobservable 
characteristics, such as student motivation, parental encouragement, and other factors, unless 
these characteristics are strongly correlated with observable measures included in the 
statistical models. This is the problem of self-selection, namely that participation in an 
intervention and performance in that intervention may be accounted for by the unmeasured 
characteristics of participants rather than the program intervention. Thus, while the outcomes 
reported here are suggestive, they are not conclusive. It is possible that other factors not 
included in the models account for both participation in an LLCP and the differences in 
outcomes. Readers should keep this limitation in mind when evaluating the evidence of the 
efficacy of the LLCPs reported here. 


7 https:/ / nces.ed.gov/ ccd/ 

8 The six districts are Antioch Unified School District, Long Beach Unified School District, Oakland 
Unified School District, Pasadena Unified School District, Porterville Unified School District, and 
West Contra Costa Unified School District. 

9 PSM was described by Rosenbaum and Rubin in their 1983 article, “The Central Role of the 
Propensity Score in Observational Studies for Causal Effects.” Riometrika , 70: 41-55. 
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There were a number of outcomes with high levels of missing data; for some measures all 
information from an LLDI district was missing for a particular cohort and in others, 
information was missing for large proportions of the comparison group. No substitution or 
imputation for missing data was done for model covariates or outcome variables. Some 
models had high rates of missing data resulting from a missing variable at the district-level. In 
such instances, results are presented both including and excluding the district in question. The 
gaps resulting from missing data may affect some of the results reported here. In addition, 
students in the LLCPs and the matched comparison were not necessarily enrolled in the 
same high schools, 10 so participation in an LLCP may be confounded with school quality 
(for example, if LLCP students attended better schools than did comparison group 
students). Further, three of the LLCPs (in the Long Beach Unified School District) may use 
grade-eight GPA as an admissions requirement. * 11 To the degree that GPA is unrelated to 
other measures included in the statistical models used in this report, results for these three 
LLCPs may be confounded with students’ preexisting academic ability. 

This report shows the outcomes for students who participated in an LLCP, defined as 
having been enrolled in the same LLCP in both grades 11 and 12 in the same high school, 
compared with a matched comparison group of similar students (on observable 
characteristics) who were not enrolled in any pathway. This study excludes students enrolled 
in any Linked Learning pathway that had not been certified, as well as those students 
enrolled in an LLCP for only one year (e.g., grade 11 or grade 12). Including such students in 
the study would confound results, because these students fail to meet the definition of either 
an LLPC student or a student not enrolled in any pathway. Both LLCP and matched 
comparison groups were restricted to those who had valid pre-high school (grade eight) 
scores on the CST mathematics and English assessments. Outcomes include attendance, 
suspension, high school graduation, progress toward completing the admissions 
requirements for the University of California (UC) and the California State University (CSU) 
systems, and postsecondary enrollment and persistence. 12 


10 This is more likely for comparison group students than for LLCP students. The percentage of 
LLCP students enrolled in high schools that also provided comparison group students in the three 
cohorts were 91 percent, 64 percent, and 67 percent for the AY 2010-11, AY 2011-12, and AY 
2012-13 cohorts. The equivalent percentages for comparison group students were 30 percent, 18 
percent, and 1 7 percent, respectively. 

11 Only one of the three, the Community of Musicians, Performers, Artists, and Social Scientists 
(COMPASS) requires a minimum GPA (2.5). The other two may use a “recommended” GPA as one 
of the entrance requirements. 

12 In this report, differences are determined by the results of Student’s /-tests or other statistical tests 
in which the probability of a type 2 error, or the alpha level, is generally set at p < .05. Occasionally 
the text notes differences that approach but do not reach that threshold of significance but are 
substantively noteworthy. 
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OUTCOMES FOR STUDENTS PARTICIPATING IN LINKED 
LEARNING CERTIFIED PATHWAYS 

Propensity score matching and regression adjustment 13 for differences in baseline 
characteristics between the LLCP and the matched comparison group that fell between .05 
and .25 standard deviations were used to compare the two groups of grade- 12 students 
(those in an LLCP and those who were not) in three AY cohorts. Among the highlights of 
the analyses conducted for this report are the following: 

• There were mixed findings for the association between participation in an LLCP 
and measures of student engagement as measured by attendance and disciplinary 
events in grade 12. LLCP students had better attendance in AY 2011—12, but there 
was no difference between these students and the matched comparison group of 
students in 2010—11 or 2012—13. AY 2011—12 LLCP students were less likely to be 
suspended during their senior year. Although LLCP students appeared to be less 
likely to be suspended compared to their matched comparison group peers in 

AY 2012—13, the difference was not statistically significant at the .05 level but it was 
at the .08 level (tables 4 and 5). 

• Grade- 12 students in all three cohorts who participated in an LLCP had a higher 
probability of graduating from high school than students in the matched 
comparison group. However, both LLPC and matched comparison groups had 
relatively high graduation rates (89.5 and 89.3 percent, respectively) (table 6). 

• Completion of the a— g admissions requirements, a set of 15 courses and a related 
grade point average (GPA) required for admission to the UC and CSU public 
university systems, were only available for the AY 2011—12 and AY 2012—13 
cohorts. Although districts were supposed to provide transcripts for all high school 
students to assess their preparation for admission to a California public university, 
there were large proportions of missing information for two districts (Long Beach 
Unified and Oakland Unified). Analyses including these two districts showed that 
grade- 12 LLCP students in the AY 2011—12 cohort were more likely to complete 
the course work and GPA requirements for admission to both the UC and CSU 
than matched comparison group students, while among AY 2012—13 students, 


13 Regression adjustment was used for any characteristic of LLCP and comparison group students that 
differed by more than .05 standard deviation but less than or equal to .25 standard deviation at 
baseline to conform to the What Works Clearinghouse (2014) standards. After matching on the 
propensity score, the matched sample was used to assess the association between participation in an 
LLCP and the outcome of interest by regressing the outcome on the variables requiring adjustment 
along with a variable that indicated participation in an LLCP. 
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LLCP participants were more likely to complete these requirements for admission to 
a CSU than the comparison group. Excluding the districts with high proportions of 
missing information showed that LLCP students in the AY 2011—12 cohort were 
still more likely than the comparison group to meet the CSU course and GPA 
requirements but there was no difference between the two groups in UC eligibility. 
Among the AY 2012—13 cohort, LLCP participants had a lower probability of 
completing the a— g requirements for the UC (table 7). 

• There were contradictory findings for UC and CSU grade point averages calculated 
without respect to whether a student completed the 15 courses necessary for 
admission. Although none of the four comparisons for the AY 2011—12 cohort met 
the .05 statistical significance level, all were statistically significant at the p<A level. 
Though the differences were small, the UC and CSU GPAs for the AY 2011—12 
LLCP senior cohort were higher than those of the matched comparison group 
students, regardless of whether the two districts with large proportions of missing 
information were included or excluded from the analyses. This relationship was 
reversed for students in the AY 2012—13 cohort. Again, regardless of whether the 
districts with large proportions of missing information were included or excluded, 
LLCP participants had lower GPAs than students in the comparison group (table 8). 

• Postsecondary enrollment data were only available for 2011 and 2012 graduates. 
There was no difference in the probability of enrolling in postsecondary education 
for AY 2010—11 LLCP graduates and matched comparison group students. 
However, AY 2011—12 LLCP students were more likely than comparison group 
students to have enrolled in postsecondary education. There were no differences 
between the two groups in immediate enrollment after high school graduation, a 
factor associated with successful postsecondary degree completion, nor was there 
any difference in short-term (one-year) persistence (table 9). 
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INTRODUCTION 


California is not alone in its efforts to address improvements in secondary 
schools. Transforming high schools and preparing more students for success in 
postsecondary educational and career pursuits are national issues. The 
statistics for persistently high dropout rates and low student achievement , 
including a troubling achievement gap , point to a national crisis in secondary 
education. Across the country , policy makers and educators are grappling with 
the issue of how to improve our secondary schools and better prepare young 
people for life and work in the twenty-first century. (California Department of 
Education 2010, p. 3) 


Linked Learning refers to a high school reform effort that places equal emphasis on 
preparation for college and career by combining (1) rigorous academics, (2) technical 
education, and (3) exposure to real-world work experiences, which are organized around one 
or more of 15 specific industry sectors. 1 A Linked Learning pathway may be offered in a 
variety of settings, including career academies, small schools, charter schools, or as a 
program within a comprehensive high school. As was noted in a recent report from the 
University of California at Los Angeles, “. . .pathways are shaped by existing school 
structures and capacity, local opportunities for partnerships and support, the skills and 
backgrounds of instructional staff, and much more” (Saunders et al. 2013). Regardless of the 
setting, a Linked Learning pathway has four core components: 2 

1 . Rigorous academics that prepare students to enroll at a two- or four-year California 
public college or university, in an apprenticeship, or in another postsecondary 
program after graduating from high school. 

2. Career-based learning in the classroom that delivers concrete knowledge and skills 
through a cluster of three or more courses, emphasizing the practical application of 
academic learning and preparing students for high- skill, high- wage employment. 


1 The 15 sectors are agriculture and natural resources; arts, media, and entertainment; building trades 
and construction; education, child development, and family services; energy and utilities; engineering 
and design; fashion and interior design; finance and business; health sciences and medical technology; 
hospitality, tourism, and recreation; information technology; manufacturing and product 
development; marketing, sales, and service; public services; and transportation. 

2 http:/ /linkedlearning.org/ about/ 
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3. Work-based learning in real-world workplaces via job shadowing, apprenticeships, 
internships, and professional skill-building opportunities. 

4. Personalized support services that include counseling and supplemental instruction 
in reading, writing, and mathematics to help students master the rigorous academic 
and professional skills necessary for success in college and career. 

According to the Linked Learning Alliance, there are four guiding principles for a Linked 
Learning pathway: 3 

1 . Linked Learning prepares students to succeed in college, career, and life. 

2. Linked Learning prepares high school students for a full range of post-graduation 
opportunities. 

3. Linked Learning connects academics to real-world applications. 

4. Linked Learning improves student engagement. 

The California Legislature and the California Department of Education (CDE) have shown 
increasing support for Linked Learning as a promising approach to transform California 
high schools. In 2008, then-governor Arnold Schwarzenegger signed Assembly Bill (AB) 
2648, which called for the creation of a report examining the feasibility of expanding 
multiple pathways in California schools. That report, Multiple Pathways to Student Success: 
Envisioning the New California Nigh School \ 4 was released in 2010. In 2011, AB 790 allocated 
funds for a Linked Learning pilot program and revised the education code to define a Linked 
Learning program. 5 

With hundreds of schools in California implementing pathways to improve outcomes for 
students, fidelity of implementation is likely to vary according to each district’s capacity to 
implement the Linked Learning model. In 2006, the James Irvine Foundation funded 
ConnectEd: The California Center for College and Career. 6 ConnectEd works with districts 
to create policies that support effective Linked Learning pathways, provides professional 


3 http:/ /linkedlearning.org/ about/ 

4 Executive Summary is available at https:/www. wested.org/resources/multiple-pathways-to-student- 
success / Footnote 2 of the report states: “Some educators now prefer the name ‘linked learning.’ To 
be consistent with the legislation, this report uses the original terminology in the legislation: multiple 
pathways.” 

5 http:/ /leginfo.legislature. ca.gov/ faces/billNavClient.xhtml?bill_id=201120120AB790 

6 The National Center for Innovation in Career and Technical Education (NCICTE) draws on the 
expertise of nationally recognized education researchers and practitioners from a range of 
organizations. ConnectEd is one of the team members included in the Center partnership. To 
preserve objectivity in the conduct of this study, ConnectEd was excluded from development and 
review of the study and its products. 
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development to school and district staff, and publishes guides and model curricula to assist 
districts in developing and implementing Linked Learning pathways. 7 

With support from the James Irvine Foundation, ConnectEd and its partners, the College 
and Career Academy Support Network (CCASN), the National Career Academy Coalition 
(NCAC), the National Academy Foundation (NAF), and Education Trust-West, developed 
guidance for certifying pathway design and conducting a formal review process that covers 
the following (Dayton, Laplante, and Fender 20 12): 8 

• pathway design and governance; 

• curriculum, instruction, assessment, and project- and work-based learning; 

• system supports such as school and district policies, leadership, professional 
development, qualified staff, and partnerships; and 

• use of data and evaluation of program impact. 

The certification process involves an intensive review of 40 criteria, which are shown in 
appendix H. Information about outcomes for students who participated in a Linked 
Learning certified pathway (LLCP) provides a starting point for evaluating the effectiveness 
of the Linked Learning approach, because certification likely reflects a higher degree of 
fidelity to the Linked Learning model than noncertified pathways. Data from the Linked 
Learning District Initiative (LLDI), which includes information on students who participate 
in an LLCP and students who do not, allow for such an evaluation. 


THE LINKED LEARNING DISTRICT INITIATIVE 

In 2009, the James Irvine Foundation funded the LLDI to “(provide) financial support, 
technical assistance, and coaching to help nine medium and large, rural and urban school 
districts in California to plan and implement systems of high-quality pathways that are 
accessible to any student in the district.” 9 

The nine districts participating in the LLDI are Antioch Unified School District, Long Beach 
Unified School District, Local District 4 of the Los Angeles Unified School District, 
Montebello Unified School District, Oakland Unified School District, Pasadena Unified 


7 See http://www.connectedcalifornia.org/ curriculum for sample curricula for pathways. 

8 There are 37 LLCPs in California. The certification process is currently under review and may be 
changed in the future. 

9 http://www.connectedcalifornia.org/ schools_districts/ certified_linked_learning_pathways 
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School District, Porterville Unified School District, Sacramento City Unified School District, 
and West Contra Costa Unified School District. 

The LLDI includes technical assistance provided by ConnectEd and the Los Angeles Small 
Schools Center (now the Center for Powerful Public Schools), Stanford University’s Center 
for Opportunity Policy in Education, San Diego State University’s School of Education, 
CCASN, and NAF . 10 Other organizations supporting the LLDI include the Institute for 
Evidence Based Change (IEBC), which is developing a database and an online tool for 
schools and districts to review information about student progress, and SRI International, 
which is conducting an overall evaluation of the Initiative . * 11 

Each of the LLDI-participating districts provides information for all students in their 
respective high schools about students’ academic and demographic characteristics, 
attendance, coursetaking, scores on state assessments, progress toward meeting the entrance 
requirements to California’s public universities, and a measure of participation in an LLCP . 12 
These data provide the basis for the analyses reported here. 


10 Examples of technical and programmatic assistance can be found at 

http:/ / casn.berkeley.edu/ resources.php?r=497&c=; http:/ / powerfuled.org/ our services /linked- 
learning; http:/ / edpolicy.stanford.edu/ projects/193; and 
http://go.sdsu.edu/ education/ ste/ cohort_Knked_learningAn_detail.aspx? 

11 A description of the LLDI can be found at http://www.connectedcalifornia.org/ 
schools_districts/ district_initiative. SRTs most recent interim report can be found 

at https://www.sri.com/ work/ publications/ taking-stock-california-linked-learning-district-initiative- 
fifth-year-evaluation. 

12 Among the nine districts included in the LLDI, three districts — Montebello Unified School District, 
San Diego Unified School District, and Sacramento City Unified School District — either did not sign 
a Memorandum of Understanding allowing use of their data (San Diego and Sacramento) or did not 
have an LLCP at the time of the study (Montebello) and are not included in this report. See the Data 
and Methodology section for a detailed description of the data used in this report. 
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PURPOSE OF THIS REPORT 

Linked Learning pathways have garnered considerable attention from local, state, and private 
funding sources, 13 so there is an interest in examining how well students enrolled in these 
pathways do both while in high school and after they leave to enter the workforce or 
postsecondary education. An examination of outcomes for students enrolled in an LLCP — 
which has been assessed for how well the pathway adheres to core components and 
principles and evaluated for implementation, ongoing monitoring of student progress, and 
student, parent, school staff, and business community support — provides an ideal sample for 
this preliminary investigation. 

Throughout the report, the grade-12 AY is used to identify the three cohorts (i.e., AY 2010— 
11 grade-12 students, AY 2011-12 grade-12 students, and AY 2012-13 grade-12 students). 

This report addresses four research questions: 

1. What is the relationship between enrollment in an LLCP and on-time graduation, 
attendance rates, and disciplinary incidents during grade 12? 

2. What is the relationship between enrollment in an LLCP and completion of high 
school course requirements (a— g courses) for admission to one of California’s public 
universities (CSU and UC systems) for the graduating classes of 2012 and 2013? 

3. What is the relationship between enrollment in an LLCP and enrollment in a 
postsecondary institution for graduates of the classes of 2011 and 2012? 

4. What is the difference in these outcomes for students participating in an LLCP 
compared with outcomes for students not participating in an LLCP? 


13 The California legislature in 2011 passed AB 790, which created a Linked Learning pilot program in 
63 local education agencies (http://linkedlearning.org/about/where-linked-learning-is- 
happening/pilot-program/). In May 2014, the California Department of Education awarded $250 
million for Career Pathways Trust grants to create “sustained career pathways programs that connect 
businesses, k-12 schools, and community colleges to better prepare students for the 21st century 
workplace.” The James Irvine Foundation has supported the development and expansion of Linked 
Learning with over $163 million in grants since 2009 (personal communication from Jessica Hickok, 
grants manager, the James Irvine Foundation, July 1, 2015). See http://linkedlearning.org/linked- 
learning-alliance/ our-members/ for business and other organizations that participate in the Linked 
Learning Alliance. 
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DATA AND METHODOLOGY 


OVERVIEW OF THE DATA 

To support the LLDI, the IEBC developed a database and an online tool for the 
participating LLDI districts. Districts were asked to upload and review data on each high 
school student and to indicate whether the student was enrolled in a certified or uncertified 
Linked Learning pathway or no pathway at all. The database includes academic and 
demographic information, including course enrollments, results on statewide assessments, 
attendance, and suspension information from each district’s student information system. 

Districts also sent IEBC information on students’ eligibility for admission to the CSU and 
UC systems. The admissions data were prepared by the Transcript Evaluation Service (TES) 
of the University of California Office of the President. Each year, TES processed the 
transcripts for all students in grades nine to 12 submitted by the districts. TES staff reviewed 
the transcripts and evaluated whether each student was “on- track” to meet the required a— g 
course requirements for admission. The a— g requirements include 15 approved college- 
preparatory courses organized into seven subject matter areas labeled “a- g.” In addition, 
TES evaluated whether students met the minimum required GPA for the a— g courses. 14 

Postsecondary enrollment data were collected from the National Student Clearinghouse 
(NSC). Each LLDI district submitted a list of all grade- 12 students who were scheduled to 
graduate in June 2011 and 2012, respectively. The students were matched to the NSC 
database of postsecondary student enrollment and attainment records, and the matched 
records were entered into the IEBC database. 

Information about each student’s high school was available from the CDE website for the 
AYs included in this report. 15 Student-level records were supplemented with information on 
the teachers’ average number of years of experience. School-level averages of the percentage 
of students with different racial/ ethnic backgrounds; the percentage of the student body 
eligible to participate in the National School Lunch Program (NSLP), a measure of 
economic disadvantage; and total enrollment were appended to the student data, as was the 


14 The a-g courses are listed below in the section on findings for the Completion of University of 
California and California State University A-G Requirements. 

15 Teacher data were unavailable for AY 2012—13. Models for the AY 2012—13 cohort that included 
teacher experience used the value for the preceding AY. 
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school’s locale, a measure of urbanicity, from the Common Core of Data (CCD). The 
individual yearly data files were combined into a single analysis file, which included all 
students in the six Linked Learning districts in the study. 


THE OVERALL ANALYSIS SAMPLE 

To be included in the overall analysis sample, 16 students had to meet all the following 
eligibility criteria. They were required to have been 

• enrolled at the same high school during grade 11 and grade 12; 

• classified as a grade- 12 student in AYs 2010—11, 2011—12, or 2012—13; 

• classified as a grade- 11 student during the previous AY; 

• had a valid scale score for the CST in grade eight in mathematics or English; and 

• enrolled in the same LLCP in grades 11 and 12 if they were participants in an 
LLCP. 17 

Students who did not meet the above criteria were excluded from the analysis file. 18 

The final analysis sample included a total of 12,136 students based on three cohorts of high 
school seniors scheduled to graduate as members of the classes of 2011 (2,719 students), 2012 
(4,824 students), and 2013 (4,593 students). Each cohort was analyzed separately. The cohorts, 
as mentioned earlier, are referred to using the appropriate AY (i.e., AY 2010—11 grade- 12 
cohort, AY 2011—12 grade- 12 cohort, and AY 2012—13 grade- 12 cohort) throughout the 
remainder of the report. 


16 The term “overall analysis sample” is used to differentiate the pool of LLCP and comparison group 
students from the subset of these students used in the analytic samples created by propensity score 
matching. 

17 The LLDI had participating districts return a “Custom File” that contained one record for each 
grade-nine to grade-12 student that indicated which, if any, pathway the student attended. Each 
pathway code was further linked to a “pathways type and model” file that provided a pathway type 
variable that distinguished Linked Learning certified pathways from other types of pathways. The data 
files prepared by IEBC for use in this study included a variable derived from the pathway type 
indicating that the student was enrolled in an LLCP or in a noncertified pathway or was not enrolled 
in any pathway. This variable was used along with an indicator of which specific LLCP the student 
was enrolled in to determine who was a Linked Learning certified pathway participant. 

18 The Tinker! Learning sample represents 74 percent of all Tinker! Learning students in the six LLDI 
districts included in the study. 
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Table 1 presents the number and percentage of students in the LLCP and comparison groups 
for each grade-12 cohort year. Table 2 presents the number and percentage of students in each 
LLCP for all three cohorts combined and separately for each grade- 12 cohort. 


Table 1: Number and percentage of students in the Linked Learning certified pathway and comparison 
groups in the overall sample for the AY 2010-11, AY 2011-12, and AY 2012-13 cohorts 






Grade-12 cohort year 




AY 2010- 

-lT 

AY 2011- 

-12 

AY 2012- 

-13 

All 

years 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

12,136 

2,719 

100.0 

4,824 

100.0 

4,593 

100.0 

Comparison 

9,586 

2,091 

76.9 

3,924 

81.3 

3,571 

77.7 

Linked Learning 

2,550 

628 

23.1 

900 

18.7 

1,022 

22.3 


a Data for Long Beach, Pasadena, and Porterville only. Other districts did not have a Linked Learning certified pathway in 
academic year (AY) 2010-11. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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Table 2: Number and percentage distribution of students in the overall sample who were in Linked Learning certified pathways, by type of pathway for the 
AY 2010-11, AY 2011-12, and AY 2012-13 cohorts 






Grade-12 cohort year 





All cohorts 

AY 2010 

-ll a 

AY 2011- 

-12 

AY 2012- 

-13 

Linked Learning certified pathway 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total 

2,550 

100.0 

628 

100.0 

900 

100.0 

1,022 

100.0 

Antioch Unified Schools 

Dozier-Libbey Medical High School b 

226 

8.9 

0 

0.0 

116 

12.9 

110 

10.8 

Long Beach Unified Schools 

Architecture, Construction, & Engineering (ACE) 

141 

5.5 

52 

8.3 

36 

4.0 

53 

5.2 

CA Academy of Mathematics and Science (CAMS) 

143 

5.6 

56 

8.9 

39 

4.3 

48 

4.7 

Community of Musicians, Performers, Artists, and 

Social Scientists (COMPASS) 

399 

15.7 

135 

21.5 

131 

14.6 

133 

13.0 

Personal Success through Empowerment, Academic Achievement, 

Conflict Resolution, and Ethics in Action (PEACE) 

477 

18.7 

160 

25.5 

154 

17.1 

163 

16.0 

Pasadena Unified Schools 

Arts, Entertainment, and Media Academy (AEM) 

157 

6.2 

44 

7.0 

45 

5.0 

68 

6.7 

Business and Entrepreneurship Academy 

141 

5.5 

35 

5.6 

63 

7.0 

43 

4.2 

Creative Arts, Media, and Design Pathway (CAMAD) 

151 

5.9 

48 

7.6 

41 

4.6 

62 

6.1 

Engineering and Environmental Science Academy (EESA) 

136 

5.3 

58 

9.2 

21 

2.3 

57 

5.6 

Porterville Unified Schools 

Partnership Academy of Business (PAB) 

68 

2.7 

23 

3.7 

20 

2.2 

25 

2.5 

Academy of Engineering 6 

13 

0.5 

0 

0.0 

4 

0.4 

9 

0.9 

Health Careers Academy 

59 

2.3 

12 

1.9 

20 

2.2 

27 

2.6 

Academy of Performing Arts 6 

5 

0.2 

0 

0.0 

1 

0.1 

4 

0.4 

Digital Design and Communications (DDC) c 

1 

0.0 

1 

0.2 

0 

0.0 

0 

0.0 

Multimedia and Technology Academy (MTA) 6 

24 

0.9 

0 

0.0 

14 

1.6 

6 

0.6 

Oakland Unified Schools 

Skyline Education Academy 6 

51 

2.0 

0 

0.0 

10 

1.1 

41 

4.0 

Life Academy of Health & Bioscience 6 

81 

3.2 

0 

0.0 

37 

4.1 

44 

4.3 

West Contra Costa Unified Schools 

Engineering Academy 6 

66 

2.6 

0 

0.0 

28 

3.1 

38 

3.7 

Law Academy 6 

78 

3.1 

0 

0.0 

39 

4.3 

39 

3.8 

Multimedia Academy 

133 

5.2 

4 

0.6 

81 

9.0 

52 

5.1 


a Data for Long Beach, Pasadena, and Porterville only. Other districts did not have a Linked Learning certified pathway in academic year (AY) 2010-11. 
b Pathway was not available in AY 2010-11. 
c Pathway was not available in AY 2011-12 or AY 2012-13. 

NOTE: Detail may not sum to totals due to rounding. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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In addition to the measures described previously, the following additional information about 
each student was included in the analysis files: 

• Demographic characteristics for each student, such as 

• sex, race/ ethnicity, ELL status in high school, and disability status; 

• NSLP eligibility; 

• whether the student had an individualized education program (IEP) prepared 
according to the requirements of the Individuals with Disability Education Act 
(IDEA ); 19 and 

• parents’ highest education level. 

• Student academic information, such as 

• proficiency levels based on scale scores for the grade-eight CST mathematics 
and English tests; 

• the number of days during AYs that the student was suspended; 

• whether the student graduated high school and the type of credential earned; 

• whether the student successfully completed the a— g course requirements for 
admission to CSU system schools ; 20 

• a measure as to whether the student successfully completed the a— g course 
requirements for admission to UC system schools; 

• the student’s GPA based on a— g courses required for admission to CSU system 
schools; and 

• the student’s GPA based on a— g courses required for admission to UC system 
schools. 


19 Under section 602(14) of the IDEA, the term “individualized education program” or “IEP” means 
a written statement for each child with a disability that is developed, reviewed, and revised in 
accordance with section 614(d) of IDEA. 

20 The a-g course requirements vary for the UC and CSU systems, as does the required minimum 
GPA. For UC admissions, a GPA of 3.0 in a-g courses is the standard, while a GPA of 2.0 in a-g 
courses is required for CSU admissions. 
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Data from the CDE provided school-level variables for each high school in LLDI districts, 
such as 

• the average number of years of teaching experience; 

• the percentage of the student population who were NSLP-eligible; and 

• the sex, race/ ethnicity, and total enrollment of the student population. 

Data from the CCD included the school’s location (in an urban, suburban, town, or rural 
area). 21 

Derived variables were created to identify characteristics about the postsecondary institution 
at which students were first enrolled after high school. These derived variables included 
whether the student enrolled immediately after high school, more than one year after high 
school, or had not enrolled as of the last date the NSC provided match data. 

Characteristics of the Samples 

Table 3 shows the distributions of student demographic and academic characteristics for the 
overall samples and selected characteristics of the schools that students attended according 
to whether they were in the LLCP or comparison group. 22 


21 The “urban-centric” locale codes as shown in https://nces.ed.gov/ccd/rural_locales.asp were 
recoded. See appendix C. 

22 The overall samples included students from 24, 54, and 47 high schools in the AY 2010-11, AY 
2011-12, and AY 2012-13 grade- 12 cohorts, respectively. 
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Table 3: Demographic and academic characteristics of students and selected school-level characteristics for the AY 2010-11, AY 2011-12, 
and AY 2012-13 cohorts overall (prior to matching) 


Grade-12 cohort year 

AY 2010-ll a AY 2011-12 AY 2012-13 

Linked Linked Linked 


Student-level characteristics 

Learning 

Comparison 

t -value 

Learning 

Comparison 

t -value 

Learning 

Comparison 

t -value 

Sex 

Male 

45.1 

49.6 

2.01 

47.2 

50.6 

1.83 

46.4 

51.4 

2.82 

Female 

54.9 

50.4 

-2.01 

52.8 

49.4 

-1.83 

53.6 

48.6 

-2.82 

Race/ethnicity 

Asian 

7.6 

14.6 

4.75 

9.2 

14.1 

3.90 

8.5 

14.5 

5.01 

Black 

14.2 

21.0 

3.99 

12.9 

17.8 

3.56 

11.3 

17.8 

5.01 

Hispanic 

58.3 

43.2 

-7.12 

59.4 

46.0 

-7.33 

62.9 

45.8 

-9.74 

White 

17.0 

18.7 

0.98 

16.7 

19.2 

1.75 

15.0 

18.7 

2.75 

Other b 

2.9 

2.5 

-0.50 

1.8 

2.9 

1.92 

2.4 

3.2 

1.35 

English Language Learner 

26.0 

35.4 

4.44 

24.7 

22.0 

-1.75 

39.1 

27.1 

-7.48 

Percent eligible for National School Lunch Program 

68.2 

67.8 

-0.16 

70.1 

64.3 

-3.31 

73.4 

66.7 

-4.03 

Ever identified as student with disability 0 

8.1 

10.9 

2.02 

8.0 

13.3 

4.34 

10.7 

16.0 

3.95 

CST math scale score 

344.8 

331.8 

-4.65 

339.3 

332.3 

-2.82 

346.4 

332.5 

-5.72 

CST English scale score 

345.8 

337.5 

-3.28 

343.0 

340.0 

-1.48 

351.5 

340.6 

-5.21 

CST math proficiency d 

Far below basic 

4.3 

8.1 

3.22 

6.4 

10.6 

3.80 

6.8 

10.8 

3.79 

Below basic 

18.0 

26.2 

4.22 

20.1 

25.2 

3.20 

18.8 

26.5 

5.06 

Basic 

36.8 

30.7 

-2.87 

31.7 

29.7 

-2.41 

28.7 

27.0 

-1.08 

Proficient 

32.5 

27.0 

-2.21 

32.7 

26.0 

-4.03 

34.0 

24.3 

-6.17 

Advanced 

9.1 

7.7 

-1.16 

9.0 

10.4 

1.25 

11.6 

11.2 

0.39 

CST English proficiency d 

Far below basic 

5.7 

10.4 

3.55 

7.7 

11.0 

2.97 

6.4 

10.6 

4.04 

Below basic 

11.0 

18.1 

4.22 

11.1 

13.9 

2.25 

10.8 

15.8 

4.03 

Basic 

38.5 

31.6 

-3.26 

35.8 

30.2 

-3.26 

32.7 

30.0 

-1.65 

Proficient 

30.3 

23.7 

-3.31 

29.3 

27.4 

-1.14 

28.6 

24.8 

-2.47 

Advanced 

14.5 

16.2 

1.04 

16.1 

17.4 

0.91 

21.5 

18.6 

-2.08 

CST math subject 

47.0 

50.1 

1.37 

61.2 

62.4 

0.70 

66.8 

67.0 

0.12 


See notes at end of table. 
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Table 3: Demographic and academic characteristics of students and selected school-level characteristics for the AY 2010-11, AY 2011-12, 
and AY 2012-13 cohorts overall (prior to matching)— Continued 


Grade-12 cohort year 

AY 2010-ll a AY 2011-12 AY 2012-13 

Linked Linked Linked 


School-level characteristics 

Learning 

Comparison 

t-value 

Learning 

Comparison 

t-value 

Learning 

Comparison 

t-value 

Percent minority students 

73.5 

67.9 

-10.86 

73.9 

66.5 

-16.24 

74.6 

67.1 

-17.02 

Percent eligible for National School Lunch Program 

67.9 

64.3 

-5.94 

65.9 

62.9 

-5.58 

67.9 

60.9 

-7.72 

Average number of years of teacher experience 6 

13.1 

14.5 

25.36 

13.8 

14.6 

11.94 

— 

— 

— 

Locale 

Urban 

99.8 

88.1 

-9.07 

70.1 

61.0 

-5.09 

75.3 

63.4 

-7.13 

Suburban 

0.0 

5.0 

5.09 

17.0 

33.5 

9.80 

13.9 

31.9 

11.46 

Town or rural 

0.2 

7.9 

7.19 

12.9 

5.5 

-8.02 

10.8 

4.7 

-7.20 

School District 

Antioch Unified f 

— 

— 

— 

12.9 

16.6 

2.78 

10.8 

17.0 

4.83 

Long Beach Unified 

64.2 

59.5 

-2.08 

40.0 

24.7 

-9.33 

38.8 

25.8 

-8.15 

Oakland Unified f 

— 

— 

— 

5.2 

15.4 

8.09 

8.3 

13.7 

4.63 

Pasadena Unified 

29.5 

23.6 

-2.97 

18.9 

16.5 

-1.71 

22.5 

17.8 

-3.37 

Porterville Unified 

6.4 

16.8 

6.60 

6.6 

10.0 

3.24 

6.9 

10.8 

3.66 

West Contra Costa Unified f 

— 

— 

— 

16.4 

16.7 

0.20 

12.6 

14.8 

1.72 


—Not available. 

a Data for Long Beach, Pasadena, and Porterville only. Other districts did not have a Linked Learning certified pathway in academic year (AY) 2010-11. 
b Other includes American Indian, Pacific Islander, Filipino, and two or more races. 
c A student identified with a disability under the Individuals With Disabilities Education Act 

d Proficiency levels are based on scale scores for the grade-eight CST mathematics tests and grade-eight CST English tests. 
e Teacher data were unavailable for AY 2012-13. 
f District had no LLCP in AY 2010-11. 

NOTE: Detail may not sum to 100 due to rounding. Statistical significance determined by a Student's t- test. The t-value is the difference between two estimates divided by the standard error of the 
difference. T-values whose absolute values are greater than or equal to 1.96 are statistically significant at the p<.05 level. In this table CST means the California Standards Test. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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There were differences between the unadjusted characteristics of LLCP students and 
students in the comparison group (those without any pathway experience) in the overall 
samples for the three grade-12 cohorts. Generally, LLCPs had higher percentages of female 
students. 23 They also had greater percentages of Hispanic students and lower percentages of 
Asians and African-Americans. LLCP participants were more likely to be NSLP- eligible, a 
measure of economic disadvantage. Although there was a higher percentage of ELL students 
among the comparison group in the AY 2010—11 cohort (35 versus 26 percent), higher 
percentages of ELL students were in the treatment group in the AY 2011—12 and AY 2012—13 
cohorts (25 versus 22 percent and 39 versus 27 percent). 

The AY 2010—11 and AY 2012—13 grade-12 LLCP students had higher grade-eight CST 
scale scores in English and mathematics. In addition, higher percentages of these students 
were rated proficient than comparison group students. 

LLCP students attended schools that differed in several noticeable ways from schools that 
comparison group students attended. LLCP students attended schools with higher 
percentages of African-American, Hispanic, and Native American students; had higher 
percentages of NSLP- eligible students; had teachers with fewer years of teaching experience; 
and were more likely to attend high schools in urban areas. 

The wide range of differences between students enrolled in an LLCP and comparison group 
students required an analytic approach that could create groups better matched on 
observable differences at baseline before program effects were assessed. PSM was used to 
create these better matched samples to reduce the possibility that any preexisting differences 
between LLCP and comparison group students were mistaken for effects of participation in 
an LLCP. 


DATA QUALITY AND DATA LIMITATIONS 


While IEBC serves as a custodian for the LLDI data, LLDI districts provided the student 
data to IEBC. 24 Given that districts have varying interest, capacity, and resources to devote 
to submitting and reviewing their data, the quality and completeness of the data may vary 
from district to district and from one cohort to the next. For some measures, data were 


23 In this report, differences are determined by the results of Student’s /-tests or other statistical tests 
in which the probability of a type-2 error, or the alpha level, is generally set at .05. Occasionally the 
text notes differences that approach but do not reach that threshold of significance but are 
substantively noteworthy. 

24 This is changing as more information becomes available from California’s state longitudinal data 
system (CALPADS). 
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unavailable because the source of the information was a non-LLDI district for which there 
was no data-sharing agreement. For example, if a student transferred to an LLDI district 
from a private school or a public school in a non-LLDI district after grade eight, the LLDI 
district might have only limited data on the student prior to the transfer. This was the case 
for grade-eight CST data for Porterville. 

Other information may be unavailable due to when an LLCP was certified. Data for 
AY 2010—11 included only three of the six LLDI districts because three districts did not 
have an LLCP in this year. Thus, the AY 2010—11 senior cohort includes approximately half 
the number of cases as the AY 2011-12 grade-12 and AY 2012-13 grade-12 cohorts. 

Occasionally, data for an entire district are missing for one or more cohorts. Pasadena, for 
example, has no NSC data for the two cohorts (AY 2010—11 and AY 2011—12) used to 
examine postsecondary outcomes. Other districts are missing information on attendance, 
suspensions or TES outcomes for all students or particular groups of students. To ensure 
that readers are aware of which groups may or may not have been included in a particular 
analysis, notes to each table specify the appropriate groups. 

The absence of grade-eight assessment data was considered especially important, because 
grade-eight assessments were the only measure of pre-high school academic achievement. 
Therefore, in addition to requiring that students had to have attended the same high school 
during their junior and senior years, it was also required that students have grade-eight 
assessment results. 

No substitution or imputation for missing data was done for covariates or outcome 


measures. 
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STATISTICAL APPROACH-MATCHING 


PROPENSITY SCORE MATCHING 

A randomized controlled trial (RCT) creates groups that are, in expectation, equivalent on 
both measured and unmeasured characteristics, although even in an RCT, there may be 
differences between the treatment and comparison groups on some covariates. Such 
differences are not systematic, however, and reflect random error. 

The goal of matching methods is to create groups as similar as possible to those created by 
an RCT, given observational data. If the two groups are equal on all measured and 
unmeasured factors, any differences in outcomes could be ascribed to the treatment that one 
group received and the other did not. Unless unmeasured factors are highly correlated with 
measured attributes, potential bias is always a possibility that places restrictions on drawing 
causal conclusions. Unlike an RCT, analysts using observational data cannot assign subjects 
to treatment, so the possibility that treatment assignment and the outcome are correlated 
requires attention. 

The voluntary nature of student participation in an LLCP means that any analysis of the 
relationship between participation in a certified pathway and student outcomes needs to 
address selection bias. For example, students who enroll in an LLCP may differ from other 
students according to a variety of observed and unobserved characteristics that affect their 
participation in an LLCP and performance on the outcomes of interest. This study uses 
propensity score matching (PSM), which provides a framework that allows for strong 
inferences to be drawn from observational data. 25 

A propensity score is the probability that a subject will receive treatment, conditional on 
measured covariates. Let X represent the vector of covariates for each subject, Z a treatment 
indicator (1= treated, 0=control), then the propensity score, e(X), is 

e(X) = prob(Z = 1|X). 

The propensity score provides a single measure that can represent any number of measured 
covariates and overcome the “curse of dimensionality” in matching. For example, if one 
wanted to match two groups on eight different measures, and each measure was a “yes/no” 


25 PSM was first described by Rosenbaum and Rubin (1983). For a comprehensive presentation of the 
key assumptions of matching methods, see Guo and Fraser (2015) and Stuart (2010). 
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dummy variable, there are 2 8 or 256 different categories on which subjects need to be 
matched. Given a large number of covariates, even with a large sample, the likelihood of 
finding subjects for each category is small. Rosenbaum and Rubin (1983) showed that 
matching on the propensity score creates groups that have the same average distributions on 
covariates as they would have if the individual covariates had been used for the matching. 
Instead of having to match eight variables, propensity score analysis requires matching on 
only one measure, the propensity score. 26 

PSM matches LLCP students with comparison group students who have similar propensity 
scores, but did not enroll in an LLCP. Effective matching often results in a group of LLCP 
students that is similar to the group of comparison students across those characteristics used 
to estimate the propensity scores. Without employing a statistical method, such as PSM, it 
would not be appropriate to ascribe differences in academic outcomes on whether or not a 
student enrolled in an LLCP. The steps of the matching process used in the present analysis 
are described in the following section. 

Analytic Approach 

The analytic approach begins by estimating a propensity score using logistic regression. The 
true propensity score is unknown and needs to be estimated. Generally, logit or probit 
models are used to estimate the conditional probability of treatment assignment. 27 The 
covariates selected for the model are included because they are suspected of affecting either 
the likelihood of treatment or the outcome. Although the choice of covariates may be 
influenced by the known association between a covariate and the outcome of interest, the 
outcome does not play a part in estimating the propensity score. One of the strengths of 
propensity score analysis is that there is a separation between the creation of the propensity 
score and the analysis of the outcome, similar to an RCT where assignment to treatment is 
independent of the outcome. Variables included in the propensity score model may include 
measures that are constant over time (gender, race/ ethnicity) or occur before the treatment 
(grade-eight test scores). The model must not include variables that could be affected by the 
outcome. 


26 As Joffe and Rosenbaum (1999, p. 328) note, “the balance on the observed covariates X that is 
obtained by matching or stratifying on an estimated propensity score is of course imperfect, but it is 
typically somewhat better than the balance on X obtained by random assignment of treatments.” 

27 Logit and probit are nonlinear statistical methods for analyzing qualitative dependent variables, 
usually a variable with only two categories. Both methods generally provide similar estimates but 
differ in how the error term is distributed (for pro bit, normal distribution with mean 0, variance 1 ; for 
logit, logistic distribution with mean 0, variance=7T 2 /3). 
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The models estimated for this report use the estimated logit in place of the estimated 
probability because the logit has an approximately normal distribution. 28 

Unless there is overlap in the estimated propensity scores, some LLCP students will not have 
a matched comparison and will need to be dropped from the analysis. The overlap is 
illustrated using histograms showing the distribution of estimated propensities for the LLCP 
and comparison group students before and after matching and report the number of LLCP 
cases that were removed from the analytic samples for overlap failure. 

Second, after estimating the propensity score, a matching algorithm is applied. In this report, 
nearest-neighbor caliper matching on the propensity score is used. Nearest-neighbor 
matching selects a treated subject and finds the comparison group subject or subjects with 
the closest set of covariate scores to the treated subject. Nearest-neighbor matching can lead 
to bad matches if the nearest comparison group score is far from the treated subject’s. To 
avoid this, a caliper (.25 times the standard deviation — or .25*SD — of the propensity score 
logit) minimizes bad matches. 29 To increase the efficiency of the estimates, two or more 
comparison group students were matched to each LLCP student (Smith 1997). 30 

Appendix D shows the estimated coefficients for the logit models used to generate the 
propensity scores for the three grade- 12 cohorts. The tables show the covariates used to 
estimate participation in an LLCP, such as gender, race/ ethnicity, parent education, whether 
the student was classified as ELL in high school, eligibility for participation in NSLP, and 
whether the student ever had a disability. Statewide assessments in mathematics and English 
administered in grade eight are also included. While the English assessment is the same for 
all grade-eight students, grade-eight students take a variety of mathematics courses (e.g., 
Algebra I or geometry in place of general math). To allow for different levels of course 
difficulty, the type of mathematics subject was dichotomized into standard and advanced, 
which included Algebra I, Algebra II, and geometry, and this variable was included in the 
estimation equation. 

Creating the propensity score models is an iterative process that involves specifying models 
with covariates suspected of affecting the outcome and the likelihood of participation in an 
LLCP, matching on the propensity score, and evaluating the post-matching covariate 
balance. Whenever balance was less than desired, the models were modified in an attempt to 
improve the balance. These modifications include creating interactions among covariates, 


28 The logit is equal to In where Pj is the probability of assignment to the treatment. 

29 Both Guo and Fraser (2015) and Rosenbaum and Rubin (1983) suggest setting the caliper to a 
quarter of the standard deviations of the propensity score. 

30 More than two matches may occur when comparison group students are tied on the estimated 
propensity score. 
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transforming some (for example, using the natural logarithm of continuous variables such as 
total school enrollment) or including quadratic terms. Thus, some logit models include 
interaction effects for combinations of proficiency and type of math course assessed or 
proficiency and scale scores in English. Several school-level measures, such as the percentage 
of minority students in the school, the percentage of the student body eligible for NSLP, and 
teachers’ average number of years of experience could also be included in logarithmic form. 
In two of the three models, indicator variables for district were included, while in the third 
(AY 2010—11), school locale was used, recoded as urban versus all other locations. The 
tables indicate the reference category for categorical variables. Besides the coefficients, their 
standard errors, and the statistical significance of the coefficient, the number of cases used in 
the logit analysis appears at the bottom of the table. 

PSM is used to estimate the average treatment effect on the treated (ATT), which estimates 
the average change in an outcome among LLCP students only. An alternative method for 
calculating treatment effects is the average treatment effect (ATE), which describes the 
average change in an outcome among all students. Given the focus of the current study is 
the relationship between participation in an LLCP and academic outcomes, the ATT is the 
appropriate measure of treatment effect. The user- written Stata program psmatch2 (Leuven 
and Sianesi 2003) was used to calculate the ATT. Psmatch2 also calculated normalized weights 
that represent the number of times a comparison group case was used in the matching 
(which were used in the statistical adjustment models described below). 

Since some LLCP students may have propensity scores outside the range of scores for 
comparison group students, the matched sample excludes any LLCP student whose 
propensity score is greater than the maximum and less than the minimum of scores for the 
comparison group. Requiring common support reduced the number of LLCP students in 
the matched sample by 65 (out of 565) in the AY 2010—11 grade-12 cohort and 46 (out of 
809) in AY 2011—12 grade-12 cohort; there were no LLCP cases removed from the 
AY 2012—13 grade- 12 cohort. The histograms in Appendix F show the distribution for the 
propensity scores for the LLCP and the comparison group students before and after 
matching and help demonstrate the overlap in scores for the two groups. 

Pstest, part of the Stata program psmatch2 , is used to evaluate how well the matching process 
balanced the covariates in the LLCP and comparison groups of students in the baseline 
analytic samples. Since matching may not be able to achieve satisfactory balance for every 
covariate, the What Works Clearinghouse (WWC) provided guidance for determining how 
much imbalance is acceptable and statistically adjust the estimated effects of participation in 
an LLCP for any remaining imbalance. Balance is assessed using the WWC’s absolute effect 
size difference (2014, p.15). For a continuous variable, such as an assessment scale score, the 
absolute effect size for continuous variables is equal to the absolute value of the difference in 
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the means between the LLCP students and the matched comparison group students, divided 
by the pooled standard deviation: 


{M ll - M c ) 



where Mll and Me are the means for the Linked Learning and comparison group students, 
respectively, and the S 2 are the respective variances. For categorical variables, the absolute 
effect size is based on the difference in proportions divided by the pooled standard 
deviation. 

The baseline equivalence tables (Appendix E) show the means, the absolute effect size, and 
the /-value for the difference in means for each covariate in the propensity score model after 
matching. 31 The post-matching absolute effect size represents the distribution of covariates 
for the LLCP students and the group of matched comparison students at baseline prior to 
assessing the effect of participation in an LLCP. The WWC Procedures and Standards Handbook 
(2014, p.15) exempts studies using quasi- experimental designs, such as PSM, from needing 
to adopt special measures to statistically adjust program effects for covariates with absolute 
effect size differences of less than .05. Covariates with differences greater than .05 and less 
than .25 must be included in models estimating program outcomes. This study uses a 
weighted regression adjustment to estimate differences in outcomes for LLCP students — 
logistic regression for binary outcomes and ordinary least squares (OLS) regression for 
continuous outcomes — when any of the covariates in the propensity score matched sample 
had absolute effect sizes requiring adjustment. The weights used in the regression models are 
calculated by psmatch2 and represent the number of times a case was used as a match (all 
LLCP cases have a weight of 1.0). The weights allow the regression results to represent the 
average treatment effect on the treated (ATT) for participation in an LLCP. An example of 
estimating a propensity score for an outcome and the diagnostics for PSM follows. 

Appendix table D1 shows the logistic regression results estimating participation in an LLCP 
for all three grade-12 cohorts. The columns show the results for AY 2010—11, 2011—12, and 
2012—13 cohorts. Appendix tables El, E2, and E3 show the corresponding baseline 
differences for the LLCP and comparison group students in each cohort after matching on 
the propensity scores derived from the logit models in appendix table Dl. For example, 


31 The /-test results are shown for readers who may be interested in the statistical significance of the 
remaining differences between the LLCP and matched comparison group students. However, the 
significance of these differences is affected by sample size, which is generally reduced by matching; 
thus the WWC focuses on the absolute effect size for determining the adequacy of the matching to 
produce equal groups at baseline. 
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appendix table E3 indicates that at baseline, before any outcomes are evaluated, the 
proportion of female students in the LLCP group in the AY 2012—13 cohort is .548. The 
proportion of female students in the matched comparison group is .556. The difference 
between the proportions represents an absolute effect size difference of .016. This difference, 
however, is not statistically significant (/>=.737). Since the absolute effect size is less than .05, 
gender is not included in models estimating the effect of participation in an LLCP on 
outcomes for the AY 2012—13 grade- 12 cohort. 

There are several covariates that have absolute effect size differences greater than .05 and 
less than or equal to .25 that do require statistical adjustment per the WWC procedures and 
guidelines. These are shown in bold font in the appendix E tables. For the AY 2012—13 
cohort, for example, these include total school enrollment, and indicators for students 
attending schools in Oakland and Pasadena. All three grade- 12 cohorts had one or more 
covariates that required statistical adjustment. 32 

Once the best propensity score match is created and the covariates needing adjustment are 
identified, weighted logistic or OLS regression (depending on whether the outcome was 
categorical or continuous) is used with the matched sample to estimate the difference in 
outcomes associated with participation in an LLCP. The outcome is regressed on the 
indicator for participation in an LLCP and any of the covariates whose absolute effect size 
requires their inclusion. 

The outcomes reported are marginal effects or the change in the outcome for a change in 
the covariate, and in particular, a change in the variable indicating participation in an LLCP. 
For continuous covariates the marginal effect is the OLS regression coefficient for the 
variable indicating participation in an LLCP. Marginal effects for categorical covariates, such 
as the dummy variable representing whether a student was in an LLCP or in the comparison 
group, represent discrete change as the value of the variable goes from 0 (comparison group) 
to 1 (LLCP participant). For categorical outcomes, such as graduated high school, met the 
UC and CSU a— g requirements, etc., the marginal effect represents the change in probability 
in the outcome for LLCP students compared to comparison group students. Unlike OLS 
regression, calculating marginal effects for nonlinear models such as logistic regression 
depends on the values of other variables included in the model, which requires assumptions 
about what values to use. For this study, marginal effects are estimated setting other 
covariates at their mean values. As such, the results indicate the treatment effect for a 
theoretical average LLCP student compared to the average student in the comparison group. 


32 Appendix table G1 shows the distributions for the outcome measures for the overall sample and 
matched sample used for estimating the difference associated with participation in an LLCP. 
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RESULTS 


Pathways are intended to increase student engagement, reduce high school dropout 
rates, improve student achievement, increase high school completion and postsecondary 
transition rates, and boost students' earning power after high school— in short, 
transform the high school experience and prepare students for both college and career, 
not just one or the other. (Clark et al. 2012) 


ATTENDANCE 


Students need to attend school daily to succeed. The good news . . . is that being in 
school leads to succeeding in school. Achievement, especially in math, is very 
sensitive to attendance, and absence of even two weeks during one school year 
matters. Attendance also strongly affects standardized test scores and graduation 
and dropout rates. Educators and policymakers cannot truly understand 
achievement gaps or efforts to close them without considering chronic absenteeism. 
(Balfanz and Byrnes 2012) 


Attendance has been shown to be an important predictor of high school completion, 
particularly in grade nine (Allensworth and Easton 2007). Attendance is a measure of student 
engagement, which has been linked to academic achievement. Although estimating the 
relationship between participation in an LLCP and grade-nine attendance was not feasible, 
the attendance ratio (days present to days enrolled) was available for attendance in grade 12 
for all three cohorts (table 4). 
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Table 4: Adjusted marginal effects of participation in a Linked Learning certified pathway on grade-12 
attendance (days attended as a proportion of days enrolled) for the AY 2010-11, AY 2011-12, and 
AY 2012-13 Linked Learning certified pathway and matched comparison group students 



Cohort 

Number of 

students 

Marginal 

Effect 3 

Robust 

Standard 

error 

t b 

Significance 


AY 2010-ll d,e 

1,916 

0.003 

0.003 

1.01 

0.311 

Attendance 0 

AY 2011-12 f,g 

3,534 

0.013 

0.004 

3.56 

0.006 


AY 2012-13 f ' h 

3,666 

-0.002 

0.004 

-0.47 

0.638 


a Marginal effects for estimated average treatment on treated from ordinary least squares regression of attendance on 
indicator for students participating in a Linked Learning certified pathway compared to matched comparison group 
students. 

b The t-value is the difference between two estimates divided by the standard error of the difference. 
c Ratio of days attended to days enrolled. 

d Data for Long Beach and Pasadena for academic year (AY) 2010-11. The remaining districts did not have Linked Learning 
certified pathways in AY 2010-11. 

e Estimate adjusted for race/ethnicity equals black; log percent of school enrollment of students who are black, Hispanic, 
and Native American; and log percent of school eligible for the National School Lunch Program per What Works 
Clearinghouse (2014) guidelines. 

f Includes data for Antioch, Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa for AY 2011-12 and 
AY 2012-13. 

g Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa, log enrollment, percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken and the interaction of the grade-eight mathematics subject; and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 

h Estimate adjusted for school enrollment, and district equal to Oakland or Pasadena per What Works Clearinghouse 
(2014) guidelines. 

SOURCE: Data collected by the Linked Learning District Initiative. 


There was no consistent pattern of higher attendance for LLCP students. The difference 
between LLCP students and their matched comparison peers was statistically significant in 
AY 201 1-12 but not in AY 2010-1 1 or AY 2012-13. The marginal effect of an LLCP on 
attendance was small, even when the difference was statistically significant. Based on the 
results for the AY 2011—12 cohort, LLCP students would have attended 2.3 days more than 
students in the comparison group based on a standard 1 80-day school year. 

In the future, as more data from the LLDI become available, grade-nine attendance should 
be a focus for further investigation. Many LLCPs start in grade nine, and attendance in that 
grade has been linked to academic failure and increased risk of dropping out of high school. 
By grade 12, many students who were at risk may have already dropped out so that any 
positive associations between LLCP participation and attendance may be less meaningful 
than those occurring in grade nine. 


SUSPENSION 

Attendance and suspension can be viewed as proxies for student engagement with schooling. 
Table 5 shows the difference in the probability of LLCP students and their matched peers 
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being suspended during grade 12. Recendy, suspensions and other disciplinary action have 
been the focus of statewide attention in California and nationally, particularly the 
disproportionate distribution of disciplinary events for minority students (U.S. Department 
of Education 2012). 


The LLDI dataset includes a count of the number of days a student was suspended. This 
measure was recoded to a zero/ one indicator, with one indicating a student had been 
suspended at least once in grade 12. 33 As table 5 shows, there was no difference in the 
probability of suspension for AY 2010—11 LLCP and comparison group students. However, 
AY 2011—12 grade-12 LLCP students were about 4 percent less likely (or -.04 probability) 
than matched comparison group students of being suspended. The apparent difference for 
AY 2012—13 grade-12 LLCP and comparison group students (about 1.6 percent less likely to 
have been suspended) was not significant at the .05 statistical significance level though it was 
at the .082 level. 


Table 5: Adjusted marginal effects of participation in a Linked Learning certified pathway on the probability 
of ever being suspended during grade 12 for the AY 2010-11, AY 2011-12, and AY 2012-13 cohorts 






Robust 





Number of 

Marginal 

Standard 




Cohort 

students 

Effect 3 

error 

z b 

Significance 

Ever 

AY 2010-ll d,e 

1,916 

0.001 

0.008 

0.16 

0.875 

AY 2011-12 f-8 







2,865 

-0.040 

0.012 

-3.31 

0.001 

suspended 0 

AY2012-13 m 






3,465 

-0.016 

0.009 

-1.74 

0.082 


a Marginal effects for estimated average treatment on treated from logistic regression of ever suspended on indicator of 
participation in a Linked Learning certified pathway compared to matched comparison group students. Marginal effects 
represent difference for Linked Learning certified pathway students holding covariates at their mean values. 
b The z-value is the difference between two estimates divided by the standard error of the difference with a normal 
distribution. 

c "Ever suspended" indicates that the student was suspended from school for at least one day during grade 12. 
d Data for Long Beach and Pasadena for academic year (AY) 2010-11. The remaining districts did not have Linked Learning 
certified pathways in AY 2010-11. 

e Estimate adjusted for race/ethnicity black; log percent of school enrollment of students who are black, Hispanic, and 
Native American; and log percent of school enrollment of students eligible for the National School Lunch Program per 
What Works Clearinghouse (2014) guidelines. 

f Includes data for Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa for AY 2011-12. Antioch did not 
have Linked Learning certified pathways in AY 2011-12. 

g Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken; and the interaction of the grade-eight mathematics subject and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 
h Includes data for Antioch, Long Beach, Pasadena, Porterville, and West Contra Costa. 

' Estimate adjusted for school enrollment, and district equal to Pasadena per What Works Clearinghouse (2014) guidelines. 
SOURCE: Data collected by the Linked Learning District Initiative. 


33 Students who have serious disciplinary histories may have dropped out of high school before 
grade 12, so the results reported here may not be generalizable to student behaviors occurring earlier 
in high school. 
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The reasons for these lower probabilities of suspension cannot be derived from the data 
available for this report. There are several structural characteristics of LLCPs that may be 
worthy of future research. These include the setting — an LLCP can be offered in career 
academies, small high schools, and charter schools, and they typically serve 250 to 500 
students (Atterbury 2013, p. 3). A core principle of the Linked Learning model — and one of 
the four core elements that is evaluated as part of the certification process — is student 
support, which may be easier to deliver in a smaller setting. In addition, teachers, business 
partners, and other adults involved with an LLCP may be especially interested in working 
with LLCP students, and that interest may include offering high levels of support and 
monitoring that affect the likelihood of students engaging in behaviors leading to 
suspension. Further research into the specific reasons why LLCP students appear to have 
lower rates of serious disciplinary events will require integrating observational studies along 
with systematic information from students, parents, teachers, and other school staff.- 


HIGH SCHOOL GRADUATION 


Research has shown that dropping out of high school is associated with a 
range of adverse employment and life outcomes. (1> Young people who do not 
complete high school are more likely to be unemployed ’, live in poverty , be 
dependent on welfare benefits, have poor physical and mental health, and 
engage in criminal activity than those with higher education levels. w Though 
many individuals who do not receive a high school diploma go on to earn an 
equivalency degree, such as a GED, this credential also is associated with lower 
earning potential than a traditional diploma. <2> The economic consequences of 
dropping out of high school do not stop with the individual. Society also faces 
costs in terms of greater spending on public assistance, higher crime rates, and 
lower tax revenues. (2> One study estimated that if those who dropped out of 
high school in 2011 had graduated instead, the nation's economy would 
benefit by about $154 billion over their lifetimes. 34 


The Linked Learning model emphasizes preparation for college and career, both of which 
depend on the successful completion of high school. Table 6 shows that LLCP students had 
a higher probability of graduating on time than their matched comparison group peers. The 
estimated advantage for pathway participants ranged from about 4.7 percent among 2011—12 
grade-12 students to 5.4 and 6.4 percent for AY 2010—11 and AY 2012—2013 grade-12 
students. Note that the models generating the estimated differences between LLCP students 


34 Sources for this passage may be found in Lucille Packard Foundation for Children’s Health (2014). 
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and comparison group students were estimated on samples that matched each LLCP student 
with comparison group students who had similar pre-high school academic achievement in 
grade eight, student race/ ethnicity, economic disadvantage, and other factors associated with 
successful completion of high school. 


Table 6: Adjusted marginal effects of participation in a Linked Learning certified pathway on the 
probability of high school graduation for the AY 2010-11, AY 2011-12, and AY 2012-13 cohorts 


Cohort 

Number 

of students 

Marginal 

Effect 3 

Robust 

Standard 

error 

b 

Z 

Significance 

AY 2010-ll c,d 

2,269 

0.054 

0.010 

5.62 

0.000 

AY 2011-12 e,f 

3,968 

0.047 

0.009 

5.11 

0.000 

AY 2012-13 e,g 

3,863 

0.064 

0.009 

7.23 

0.000 


a Marginal effects for estimated average treatment on treated from logistic regression of high school graduation on 
indicator of participation in a Linked Learning certified pathway compared to matched comparison group students. 
Marginal effects represent difference for Linked Learning certified pathway students holding covariates at their mean 
values. 

b The z-value is the difference between two estimates divided by the standard error of the difference with a normal 
distribution. 

c Data for Long Beach, Pasadena, and Porterville for academic year (AY) 2010-11. 

d Estimate adjusted for race/ethnicity equals black; log percent of school enrollment of students who are black, Hispanic, 
and Native American; and log percent of school eligible for the National School Lunch Program per What Works 
Clearinghouse (2014) guidelines. 

e Includes data for Antioch, Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa for AY 2011-12 and 
AY 2012-13. 

f Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken; and the interaction of the grade-eight mathematics subject and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 

g Estimate adjusted for school enrollment, and district equal to Oakland or Pasadena per What Works Clearinghouse 
(2014) guidelines. 

SOURCE: Data collected by the Linked Learning District Initiative. 


COMPLETION OF UNIVERSITY OF CALIFORNIA AND 
CALIFORNIA STATE UNIVERSITY A-G REQUIREMENTS 


Admission to the UC or the CSU system requires applicants to successfully complete a series 
of 15 yearlong courses organized into seven subject areas labeled “a- g” and meet a 
minimum GPA in these courses. 35 According to the Office of the President (2014), these 
courses include the following: 


35 Admission to a California community college does not require meeting the a-g subject or GPA 
requirements. 
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• History/ social science (“a”) - Two years, including one year of world history, 
cultures, and historical geography and one year of U.S. history, or one-half year of 
U.S. history and one-half year of American government or civics 

• English (“b”) - Tour years of college preparatory English that integrates reading of 
classic and modern literature, frequent and regular writing, and practice listening and 
speaking 

• Mathematics (“c”) - Three years of college-preparatory mathematics that includes or 
integrates the topics covered in elementary and advanced algebra and two- and 
three-dimensional geometry 

• Laboratory science (“d”) - Two years of laboratory science providing fundamental 
knowledge in at least two of the three disciplines of biology, chemistry, and physics 

• Language other than English (“e”) - Two years of the same language other than 
English or equivalent to the second-level of high school instruction 

• Visual and performing arts (“f”) - One year chosen from dance, drama/ theater, 
music, or visual art 

• College-preparatory elective (“g”) - One year chosen from the “a— f’ courses 
beyond those used to satisfy the requirements above, or courses that have been 
approved solely in the elective area 

Course content is specified by UC faculty, and high schools must receive approval before 
any course can be authorized for a— g credit. 36 In addition to completing the required a— g 
courses, students must also meet a minimum GPA in these courses (2.0 or C for the CSU 
system and 3.0 or B for the UC system). TES data, the source for a— g outcomes, were 
available for the AY 2011—12 and AY 2012—13 cohorts. 37 

Districts participating in the LLDI were asked to submit transcripts for students in all high 
schools, but districts appear to have responded to this request differendy. Long Beach, for 
example, is missing information on about two-thirds of students in both the AY 2011—12 and 
AY 2012—13 cohorts, while Oakland is missing information on over half the AY 2012—13 
cohort. Almost all of the missing TES data are for schools that had only comparison group 
students so the districts may have misunderstood the request to send all high school student 
transcripts to TES, not just transcripts for Linked Learning participants. 


36 The University of California Office of the President maintains a website that provides lists of all a-g 
courses that have been certified. The site is searchable using several criteria including district and school 
name. See https:/ /hs-articulation.ucop.edu/ agcourselist#/list/ search/ all. 

37 There were too few schools participating in TES to include the AY 2010-11 cohort. 
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The large amount and nonrandom character of this missing information for the two districts 
presents an analytic problem: Whether to remove all students from these districts or to 
assume that the comparison group students who remain in the sample, both from the two 
districts and the other four districts in the sample, are equivalent to those students whose 
information is not included. Results were calculated including and excluding Long Beach 
Unified students and Oakland Unified students. These comparisons allow readers to evaluate 
bounds on the associations between participation in an LLCP and the TES outcomes. 

There were mixed findings for participation in an LLCP and completion of the UC a— g 
requirements. In the sample that included Long Beach and Oakland, students in the 
AY 2011—12 cohort had a greater probability of completing the a— g requirements for both 
the UC and CSU. LLCP students had a 6 percent greater probability of completing the UC 
a— g requirements than comparison group students and about a 1 6 percent greater probability 
of completing the CSU a— g requirements (table 7). Excluding Long Beach and Oakland 
there was no difference between the two groups in this cohort in the probability of 
completing the UC requirements though LLCP students still had about a 13 percent greater 
probability of completing the CSU a— g requirements. 
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Table 7: Adjusted marginal effects of participation in a Linked Learning certified pathway on the 
probability of completing the University of California and California State University a-g requirements 
(both subject and GPA requirements) for AY 2011-12 and AY 2012-13 cohorts 


Including Long Beach (AY 2011- 

12, AY 2012-13) and Oakland (AY 2011-12, AY 2012-13) 






Robust 





Number of 

Marginal 

Standard 



Outcome 

Cohort 

students 

Effect 3 

error 

z b 

Significance 

UCa-g 

AY 2011-12 c,d 

2,648 

0.062 

0.028 

2.25 

0.024 

AY 2012-13 c,e 

2,578 

0.007 

0.024 

0.29 

0.771 

CSU a-g 

AY 2011-12 c,d 

2,648 

0.155 

0.034 

4.50 

0.000 







AY 2012-13 t,e 

2,578 

0.072 

0.025 

2.83 

0.005 

Excluding Long Beach (AY 2011- 

■12, AY 2012-13) and Oakland (AY 2011-12, AY 2012-13) 






Robust 





Number of 

Marginal 

Standard 



Outcome 

Cohort 

students 

Effect 3 

error 

z b 

Significance 


AY 2011-12 f,g 

2,262 

0.024 

0.030 

0.79 

0.429 

UCa-g 

AY2012-13 h,i 






1,977 

-0.066 

0.032 

-2.04 

0.041 

CSU a-g 

AY 2011-12 f,e 

2,262 

0.130 

0.036 

3.57 

0.000 

AY 2012-13 h,i 

1,977 

-0.036 

0.034 

-1.05 

0.292 



a Marginal effects for estimated average treatment on treated from logistic regression of UC/CSU a-g course 
requirements on indicator of participation in a Linked Learning certified pathway compared to matched comparison 
group students. Marginal effects represent difference for Linked Learning certified pathway students holding covariates 
at their mean values. 

b The z-value is the difference between two estimates divided by the standard error of the difference with a normal 
distribution. 

c Data for Antioch, Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa for academic year (AY) 2011-12 
and AY 2012-13. 

d Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken; and the interaction of the grade-eight mathematics subject and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 

e Estimate adjusted for school enrollment, and district equal to Oakland or Pasadena per What Works Clearinghouse 
(2014) guidelines. 

f Includes data for Antioch, Oakland, Pasadena, Porterville, and West Contra Costa for AY 2011-12. 
g Estimate adjusted for districts equal to Oakland, and West Contra Costa; log enrollment; percent of school enrollment of 
students who are black, Hispanic, and Native American squared; which grade-eight CST mathematics subject assessment 
(advanced mathematics— Algebra I, Geometry, Algebra II— or general mathematics) was taken; and the interaction of the 
grade-eight mathematics subject and the grade-eight CST mathematics scale score per What Works Clearinghouse (2014) 
guidelines. 

h Includes data for Antioch, Pasadena, Porterville, and West Contra Costa for AY 2012-13. 

' Estimate adjusted for school enrollment, and district equal to Pasadena per What Works Clearinghouse (2014) guidelines. 
SOURCE: Data collected by the Linked Learning District Initiative. 


The results for the AY 2012—13 cohort were different. When Long Beach and Oakland were 
included there was no difference between LLCP and comparison group students in UC a— g 
completions, but LLCP students had a higher probability (.072 or 7.2 percent) of completing 
the CSU a— g requirements. Thus, in both cohorts LLCP students were more likely than 
comparison group students to meet the CSU a— g requirements. 
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After removing Long Beach and Oakland students from the analytic sample, LLCP students 
in AY 2012—13 were about 7 percent less likely than students in the comparison group to 
complete the UC a— g requirements; there was no statistically significant difference between 
the two groups in the probability of completing the CSU a— g requirements. 

Table 7, in conjunction with table 6, provides conflicting evidence that LLCP students were 
better prepared for admission to a California public university than their matched 
comparison group peers. When the Long Beach students were included in the analysis for 
the AY 2011—12 cohort, LLCP participants appeared well prepared for admission to a 
California public university, since they were 6 percent more likely to complete the UC a— g 
requirements and almost 16 percent more likely to complete the CSU a— g requirements. 

For the AY 2012—13 cohort, including Long Beach and Oakland students, LLCP students 
were about 7 percent more likely than comparison students to complete the CSU a— g 
requirements; LLCP and comparison group students had the same probability of completing 
the UC a— g requirements. 

Alternatively, if Long Beach students in the AY 2011—12 cohort were not included, LLCP 
participants were no more likely than comparison group students to complete the UC a— g 
requirements, but they were 13 percent more likely to complete the CSU a— g requirements. 
When Long Beach and Oakland students in the AY 2012—13 cohort were not included, 
LLCP participants were about 7 percent less likely to complete the UC a— g requirements and 
no more likely than comparison group students to complete the CSU a— g requirements. 

These contrasting results when the two districts without complete TES data were included 
or excluded underline the importance of working closely with districts to ensure that the 
information necessary to evaluate the outcomes for participation in an LLCP are available. 
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AVERAGE GPA FOR UNIVERSITY OF CALIFORNIA AND 
CALIFORNIA STATE UNIVERSITY A-G COURSES 

The GPA analysis includes students regardless of whether or not they met the a— g subject 
requirements. TES calculates UC and CSU GPAs for all a— g courses on submitted 
transcripts. To ensure that GPAs were compared based on consistent criteria, this report 
used the TES-calculated UC and CSU GPAs rather than any high school-calculated GPAs. 
As with the overall a— g completion analyses described in the previous section, two separate 
GPA analyses were conducted for each of the AY 2011—12 and AY 2012—13 cohorts. For 
AY 2011—12 one analysis includes Long Beach students, while the second excluded them. 
Similarly, for the AY 2012—13 cohort, one analysis included students from Long Beach and 
Oakland, while the second excluded them. The results are shown in Table 8. 38 


38 These results are based on data provided by TES. There were too few schools participating in TES 
to include the AY 2010-11 cohort. 
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Table 8: Adjusted marginal effects of participation in a Linked Learning certified pathway for grade-12 
students on the average GPAs calculated for University of California and California State University a-g 
courses for the AY 2011-12 and AY 2012-13 grade-12 Linked Learning certified pathway and matched 
comparison group students 


Including Long Beach (AY 2011- 

12, AY 2012-13) and Oakland (AY 2011-12, AY 2012-13) 


Outcome 

Cohort 

Number of 

students 

Marginal 

Effect 3 

Robust 

Standard 

error 

t b 

Significance 

UC GPA 

AY 2011-12°' d 

AY 2012-13 c,e 

2,648 

2,578 

0.08 

-0.07 

0.05 

0.04 

1.83 

-1.77 

0.068 

0.076 

CSU GPA 

AY 2011-12 c,d 

2,648 

0.09 

0.05 

1.77 

0.077 

AY 2012-13°' e 

2,578 

-0.07 

0.04 

-1.77 

0.076 

Excluding Long Beach (AY 2011- 

■12, AY 2012-13) and Oakland (AY 2011-12, AY 2012-13) 


Outcome 

Cohort 

Number of 

students 

Marginal 

Effect 3 

Robust 

Standard 

error 

t b 

Significance 

UC GPA 

AY 2011-12 f,g 

2,262 

0.08 

0.05 

1.77 

0.076 

AY2012-13 f ' h 

1,977 

-0.26 

0.05 

-4.71 

0.000 

CSU GPA 

AY 2011-12 f,e 

2,262 

0.08 

0.05 

1.68 

0.093 

AY2012-13 f ' h 

1,977 

-0.26 

0.05 

-4.75 

0.000 


a Marginal effects for estimated average treatment on treated from ordinary least squares regression of UC/CSU GPA 
requirement on indicator for students participating in a Linked Learning certified pathway compared to matched 
comparison group students. 

b The t-value is the difference between two estimates divided by the standard error of the difference. 
c Data for Antioch, Oakland, Pasadena, Porterville, and West Contra Costa for academic year (AY) 2011-12 and 
AY 2012-13. 

d Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken; and the interaction of the grade-eight mathematics subject and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 

e Estimate adjusted for school enrollment, and district equal to Oakland or Pasadena per What Works Clearinghouse 
(2014) guidelines. 

f Includes data for Antioch, Oakland, Pasadena, Porterville, and West Contra Costa for AY 2011-12 and AY 2012-13. 
g Estimate adjusted for districts equal to Oakland, and West Contra Costa; log enrollment; percent of school enrollment of 
students who are black, Hispanic, and Native American squared; which grade-eight CST mathematics subject assessment 
(advanced mathematics— Algebra I, Geometry, Algebra II— or general mathematics) was taken; and the interaction of the 
grade-eight mathematics subject and the grade-eight CST mathematics scale score per What Works Clearinghouse (2014) 
guidelines. 

h Estimate adjusted for school enrollment, and district equal to Pasadena per What Works Clearinghouse (2014) 
guidelines. 

SOURCE: Data collected by the Linked Learning District Initiative. 


The estimated marginal effects for the AY 2011—12 cohort were of similar magnitude for 
both the UC and CSU GPAs regardless of whether Long Beach was included or excluded 
from the analytic sample. The apparent advantage for participation in an LLCP (about .08 
GPA point) as compared with the average GPA for comparison group students was not 
statistically significant at the .05 level but it was at the p<A level. 
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Among members of the AY 2012—13 cohort, the positive associations between LLCP 
participation and GPA were reversed in the analyses that included and excluded Long Beach 
and Oakland. When Long Beach and Oakland were included, LLCP students had average 
UC and CSU GPAs that appeared to be .07 lower than comparison group students. These 
differences were not statistically significant at the .05 level, but they were both significant at 
the ^<.08 level. When Long Beach and Oakland were excluded, LLCP students’ GPAs were 
just over one-quarter of a GPA point (-.26) lower than the comparison group’s GPAs. 

The data available for this report do not provide an explanation for why TES outcomes 
varied from one cohort to the next. PSM ensures that the two groups of students, LLCP and 
matched comparisons, are equivalent on observable characteristics, but there are other 
factors, such as conflicting demands placed on LLCP students to meet both an academic and 
technical education curriculum or changes in the context in which Linked Learning was 
delivered, among other factors that may have come into play and affected student 
performance. More detailed observational information would be helpful in investigating 
these possible differences. In addition, the large amount of missing data for TES outcomes 
and the concomitant decision to include or exclude information for students in affected 
districts also affects estimates of the association between LLCP participation and outcomes. 
The LLDI works within constraints imposed by a district’s ability and willingness to supply 
information. The differing TES outcomes for analyses, including and excluding districts with 
large amounts of missing information, demonstrate the importance assigned to persuading 
districts to supply complete information and the need for ongoing monitoring of data that 
are submitted. 
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ENROLLMENT AND PERSISTENCE IN POSTSECONDARY 
EDUCATION 

Tables 6 through 8 show that LLCP students had higher rates of high school graduation and 
mixed experiences meeting eligibility criteria for admission to a California public university 
than their matched comparison group peers. Table 9 presents information on the association 
between participation in an LLCP and several postsecondary outcomes: the probability of 
enrollment any time after high school graduation, immediate enrollment after high school 
graduation, and one-year persistence. 

Postsecondary outcomes were available for the AY 2010—11 and AY 2011—12 grade-12 
cohorts and based on information provided by NSC for Antioch, Long Beach, Porterville, and 
West Contra Costa as of spring 2013. NSC data were unavailable for Pasadena students in the 
AY 2010—11 cohort, so outcomes were limited to only two districts, Long Beach and 
Porterville, in this year. In addition, Oakland used a different source for its postsecondary 
outcomes information. Since Oakland postsecondary enrollment information was not based 
on NSC data, marginal effects were estimated with and without Oakland for the AY 2011—12 
cohort in order to provide a sense of how the results varied according to the use of a different 
source of postsecondary enrollment data for this district. 39 Oakland did not have an LLCP in 
AY 2010-11. 


39 Unlike the data for the other districts provided by NSC, for example, all of the Oakland students 
who had any postsecondary attendance were shown as entering immediately after graduating high 
school. 
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Table 9: Adjusted marginal effects of participation in a Linked Learning certified pathway for grade-12 
students on the probability of postsecondary enrollment and persistence for the AY 2010-11, AY 2011-12 
and AY 2012-13 grade-12 Linked Learning certified pathway and matched comparison group students 






Robust 





Number of 

Marginal 

Standard 



Outcome 

Cohort 

students 

Effect 3 

error 

z b 

Significance 

Any 

AY2010-ll d,e 

1,433 

-0.028 

0.03 

-0.98 

0.328 

Postsecondary 

AY 2011-12 f,e 

3,006 

0.054 

0.03 

2.08 

0.038 

enrollment 0 

AY 2011-12 h,i 

2,566 

0.061 

0.03 

2.21 

0.027 

Immediate 

enrollment 

AY2010-ll de 

1,052 

0.018 

0.03 

0.60 

0.551 

AY 2011-12 f,e 

2,151 

0.010 

0.02 

0.51 

0.607 

AY2011-12 h ' k 

1,711 

0.005 

0.02 

0.27 

0.790 


One-year 

persistence 1 AY2010-ll d ' e 1,122 0.008 0.04 0.20 0.838 


a Marginal effects for estimated average treatment on treated from logistic regression of postsecondary education 
measures indicators of participation in a Linked Learning certified pathway compared to matched comparison group 
students. Marginal effects represent difference for Linked Learning certified pathway students holding covariates at their 
mean values. 

b The z-value is the difference between two estimates divided by the standard error of the difference with a normal 
distribution. 

c "Any postsecondary enrollment" identifies students who enrolled at a postsecondary institution after leaving high 
school based on data from the National Student Clearinghouse for Antioch, Long Beach, Porterville, and West Contra 
Costa. Data for Oakland provided by a different source. 
d Data for Long Beach and Porterville for academic year (AY) 2010-11 

e Estimate adjusted for race/ethnicity equals black; log percent of school enrollment of students who are black, Hispanic, 
and Native American; and log percent of school eligible for the National School Lunch Program per What Works 
Clearinghouse (2014) guidelines. 

f Includes data for Antioch, Long Beach, Oakland, Porterville, and West Contra Costa for AY 2011-12. 
g Estimate adjusted for districts equal to Long Beach, Oakland, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight California Standards 
Test (CST) mathematics subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general 
mathematics) was taken; and the interaction of the grade-eight mathematics subject and the grade-eight CST 
mathematics scale score per What Works Clearinghouse (2014) guidelines. 
h Includes data for Antioch, Long Beach, Porterville, and West Contra Costa for AY 2011-12. 

' Estimate adjusted for districts equal to Long Beach, and West Contra Costa; log enrollment; percent of school 
enrollment of students who are black, Hispanic, and Native American squared; which grade-eight CST mathematics 
subject assessment (advanced mathematics— Algebra I, Geometry, Algebra II— or general mathematics) was taken; and 
the interaction of the grade-eight mathematics subject and the grade-eight CST mathematics scale score per What Works 
Clearinghouse (2014) guidelines. 

j Immediate enrollment: Identifies students who enrolled at a postsecondary institution during the first summer or fall 
after leaving high school based on data from the National Student Clearinghouse for Antioch, Long Beach, Porterville, and 
West Contra Costa. Data for Oakland provided by a different source. 

k Estimate adjusted for districts equal to Long Beach and West Contra Costa; log enrollment; percent school enrollment of 
students who are black, Hispanic, and Native American squared; which grade-eight CST mathematics subject assessment 
(advanced mathematics— Algebra I, Geometry, Algebra II— or general mathematics) was taken; and the interaction of the 
grade-eight mathematics subject and the grade-eight CST mathematics scale score per What Works Clearinghouse (2014) 
guidelines. 

1 "One-year persistence" identifies students who were continuously enrolled at a postsecondary institution for at least 
one year after leaving high school based on data from the National Student Clearinghouse for Long Beach and Porterville. 
SOURCE: Data collected by the Linked Learning District Initiative. 


The top panel of table 9 shows that LLCP students in the AY 2010—11 cohort were no more 
likely than their matched comparison group peers to attend a postsecondary institution by 
spring 2013. However, among AY 2011—12 students, LLCP students had a 5 or 6 percent 
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(with and without Oakland data, respectively) greater probability of enrolling in a 
postsecondary institution by spring 2013 than did comparison group students. 

Students who delay enrollment in postsecondary education do so for a variety of reasons, 
but the evidence shows that students who delay are less likely to complete a degree than 
students who enter postsecondary education immediately after high school (Horn, Cataldi, 
and Sikora 2005; Bozick and DeLuca 2005). 

The middle panel of table 9 shows the estimated difference between LLCP students and 
their matched comparison peers in the probability that grade- 12 students enrolled in a 
postsecondary institution immediately after high school graduation. For this analysis, 
immediate entry was defined as enrolling by the fall following graduation (fall 2011 for the 
AY 2010—11 cohort and fall 2012 for the AY 2011—12 cohort). There were no differences in 
immediate enrollment for LLCP and comparison group students in either cohort nor was 
there any difference whether Oakland data were included or excluded. 

The bottom panel of table 9 presents the results for persisting in postsecondary education 
one year after first enrolling. Among the AY 2010—11 students who enrolled in 
postsecondary education in fall 2011, there was no difference in postsecondary persistence 
into fall 2012 between LLCP students and their matched comparison group peers. 

In sum, participation in an LLCP was associated with higher participation rates in 
postsecondary education for the AY 2011—12 cohort, and this finding was robust whether a 
single (NSC only) or a mixed data source was used for the analysis. There was no evidence 
of any positive association between LLCP participation and immediate enrollment in 
postsecondary education or in short-term (i.e., one-year) persistence between LLCP students 
and matched comparison group students. 

Over time, these results may change as students who started their postsecondary studies 
after the cutoff date for the NSC data used in this report (spring 2013) enroll in 
postsecondary education. As the LLDI continues to collect data for new and existing 
cohorts, a more comprehensive assessment of the effects of an LLCP on the range of 
postsecondary outcomes will become available. 
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SUMMARY OF FINDINGS AND STUDY 
LIMITATIONS 


This report is a secondary analysis of data collected as part of the LLDI. The analyses 
examined outcomes for three cohorts of grade-12 students — in AYs 2010—11, 2011—12, and 
2012—13 — in six California school districts. Data for the AY 2010—11 cohort were limited to 
three of the six districts and to only two districts for the postsecondary education outcomes. 
The analysis sample included students who participated in the same LLCP in grade 1 1 and 
grade 12, and compared their outcomes with those of their peers who attended the same 
high school in grades 11 and 12 but who were not in an LLCP. As the number of certified 
pathways — and the number of high school students enrolled in an LLCP — increases, 
research on the effects of LLCPs on academic outcomes should continue. Further, because 
many Linked Learning pathways often begin in grade nine, future studies should include 
students in grade nine and track their academic progress through high school and beyond. 
Doing so will provide a more accurate picture of the effects of certified Linked Learning 
Pathways. 

PSM was employed to create groups of LLCP students and comparison group students who 
were similar on measured characteristics, such as gender, race/ ethnicity, disability, and pre- 
high school achievement, and school characteristics, such as total enrollment, the percentage 
of students eligible for NSLP, the percentage of minority students, and teacher experience. 
Although PSM can create treatment and comparison groups that are similar to each other on 
measured characteristics, PSM is not as strong as an RCT in creating groups that are similar 
on both observable and unobservable factors. Thus, while the outcomes reported here are 
suggestive, they are not conclusive. It is possible that other factors not included in the 
models account for both participation in an LLCP and the differences in outcomes. 

Differences in school quality may also have affected the findings. Students in the LLCPs and 
the matched comparison were not necessarily enrolled in the same high schools, 40 so 
participation in an LLCP may be confounded with school quality (for example, if LLCP 


40 This is more likely for comparison group students than for LLCP students. The percentage of 
LLCP students enrolled in high schools that also provided comparison group students in the three 
cohorts were 91 percent, 64 percent, and 67 percent for the AY 2010-11, AY 2011-12, and AY 
2012-13 cohorts. The equivalent percentages for comparison group students were 30 percent, 18 
percent, and 1 7 percent, respectively. 
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students attended better schools than did comparison group students). Further, three of the 
LLCPs (in the Long Beach Unified School District) may use grade-eight GPA as an 
admissions requirement. 41 To the degree that GPA is unrelated to other measures included 
in the statistical models used in this report, results for these three LLCPs may be 
confounded with students’ preexisting academic ability. 

There were a number of outcomes with high levels of missing data; for some measures all 
information from an LLDI district was missing for a particular cohort and in others, 
information was missing for large proportions of the comparison group. These gaps may 
affect some of the results reported here. In addition, the analyses do not include direct 
measures of student engagement in learning, and there are no measures of teacher 
characteristics or experience. There are no data linking students with particular teachers or 
classrooms — information that could be valuable in articulating how various aspects of an 
LLCP affects student outcomes. 

Other studies currently under way will incorporate some of the information that was 
unavailable for this study, which in combination with the results reported here, will provide a 
more in-depth portrait of student participation in an LLCP. 

With these limitations in mind, this study found some positive though inconsistent 
associations for LLCP participation on measures of student engagement, including 
attendance and disciplinary events. With respect to some of the key goals of Linked 
Learning, successful completion of high school and preparation for and enrollment in 
postsecondary education, there was mixed evidence. 42 For all three cohorts, LLCP students 
had a higher probability of completing high school. Including Long Beach in the analysis, 
LLCP students in the AY 2011—12 cohort had a higher probability of meeting overall 
UC/ CSU a— g requirements (both subject and GPA) than matched comparison group 
students. And although LLCP students in the AY 2012—13 cohort were no more likely than 
comparison group students to complete the UC a— g requirements, LLCP students were 
about 7 percent more likely to meet the CSU a— g requirements. If districts with high 
proportions of missing TES data are excluded (Long Beach and Oakland), LLCP students in 
the AY 2012—13 cohort were about 6.6 percent less likely to meet the UC a— g requirements 
than comparison students. LLCP students in the AY 2011—12 cohort were 13 percent more 
likely than comparison group students to complete the CSU a— g requirements, even after 
excluding students from Long Beach, which accounted for a large proportion of the missing 


41 Only of the three, the Community of Musicians, Performers, Artists, and Social Scientists 
(COMPASS) requires a minimum GPA (2.5). The other two may use a “recommended” GPA as one 
of the entrance requirements. 

42 This study was unable to evaluate how well LLCPs prepared students for a career, which is 
considered equally as important as preparation for postsecondary education. 
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data. The results for UC and CSU GPAs were more consistent for each cohort. Regardless 
of whether districts with large proportions of missing data were included or excluded, LLCP 
students in the AY 2011—12 cohort had higher GPAs than did comparison group students, 
though these differences were small (about a .08 GPA point). Similarly, there was a 
consistent pattern of lower GPAs for LLCP students in the AY 2012—13 cohort for both the 
UC and CSU GPA measures. For the analyses excluding the two districts with large 
proportions of missing data in the AY 2012—13 cohort, the estimated effect for LLCP 
students was lower by about a quarter of a GPA point. 

Two years of postsecondary enrollment data were examined. For the AY 2010—11 cohort, 
there was no difference between LLCP and comparison group students in the probability of 
enrolling in a postsecondary institution by spring 2013. For the AY 2012—13 cohort, 
however, LLCP students had a 5 percent greater probability of enrolling in a postsecondary 
institution by spring 2013 and this finding was robust whether or not data from Oakland, 
which reported postsecondary enrollment information from a different source than NSC, 
were used in the analysis. Other measures of postsecondary education, including immediate 
enrollment after high school graduation, and short-term (i.e., one-year) persistence, showed 
no differences between the LLCP and comparison group students. 
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IMPLICATIONS FOR POLICY AND 
PRACTICE 


This report has shown mixed results for grade- 12 students who participated in an LLCP and 
similar students who did not participate in an LLCP on measures of student engagement and 
preparation for postsecondary education, and positive differences for high school 
graduation. For one of the two years for which postsecondary enrollment data were 
available, LLCP students had a greater probability of enrolling than did comparison group 
students. There was no difference, however, in the probability of immediate enrollment or 
one-year persistence for the one cohort for which data were available (AY 2010—11). 
Educators and policymakers interested in improving high school student outcomes may find 
these results useful in evaluating the applicability of Linked Learning to their own context. 

Unlike some high school reforms, Linked Learning has a well-developed infrastructure to 
support its growth and development. The voluntary certification process for Linked 
Learning pathways provides measurable standards against which pathways can be evaluated 
for adherence to the core components and principles of the Linked Learning model. 
Certification is one way that fidelity of implementation to the model can be achieved. At the 
same time, Linked Learning pathways are not required to undergo certification, and the 
specific implementation of particular programs may differ from the model incorporated in 
the certification process. In addition, the commitment of school and district staff to the 
Linked Learning model, the resources that individual districts have at hand, and their interest 
in allocating funds to develop and sustain a Linked Learning pathway will influence the 
benefits accruing to students. These factors were not evaluated in this study, so the results 
reported here should be viewed cautiously. 

Linked Learning has gained broad support within California. The Linked Learning Alliance, 
a coalition of California educators, local education agencies, businesses, and community 
organizations supports the growth of Linked Learning in the state. 43 Both the California 
Department of Education and the California State Legislature have supported the wider 
development of the Linked Learning approach. 


43 Member organizations are listed at http:/ /linkedlearning.org/linked-learning-alliance/ our- 
members/ 
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Districts and schools interested in developing pathways, as well as maintaining and 
improving existing ones, can draw upon the technical assistance provided by ConnectEd and 
its partners. ConnectEd has developed model curricula and made them available. Whether 
ConnectEd or a similar organization that can serve as a “standard bearer” to ensure fidelity 
of implementation is necessary to realize the apparent benefits of the Linked Learning 
approach is an open question, but it is one that should be considered. Similarly, as more 
schools and districts adopt Linked Learning there is a possibility that the extensive support 
early adopters have been offered by ConnectEd may be reduced. 

This study was unable to address a number of important questions that should be part of a 
research agenda for assessing the value of Linked Learning to improve student outcomes. 
First, although the analyses conducted for this report were able to document several positive 
outcomes for participants in an LLCP, they could not identify what components of the 
LLCP are the most effective in improving these outcomes. Close, systematic observation of 
LLCPs is needed to tease out the role of different elements in the Linked Learning model. 
Second, individual LLCPs were not examined here, so the effectiveness of one type of LLCP 
compared with another remains an open question. As more students enroll in pathways, the 
number of cases will increase accordingly, permitting these comparisons to be examined 
quantitatively as well as qualitatively. Third, the “fit” of particular types of Linked Learning 
pathways for particular types of students is of interest. Educators with diverse student bodies 
that have multiple industry sectors as potential partners in their communities will want 
evidence about whether particular types of Linked Learning pathways are associated with 
better outcomes for their students. 
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APPENDIX A. LIST OF LINKED LEARNING 
CERTIFIED PATHWAYS INCLUDED IN THE 
ANALYSES BY DISTRICT 


Antioch Unified School District 

• Dozier— Libbey Medical High School 

Long Beach Unified School District 

• Architecture, Construction, and Engineering Academy (ACE at Jordan High 
School) 

• California Academy of Mathematics and Science (CAMS) 

• Community of Musicians, Performers, Artists, and Social Scientists (COMPASS at 
Millikan High School) 

• Personal Success Through Empowerment, Academic Achievement, Conflict 
Resolution, and Ethics in Action (PEACE at Millikan High School) 

Oakland Unified School District 

• Skyline Education Academy (Skyline High School) 

• Life Academy of Health and Bioscience 

Pasadena Unified School District 

• Arts, Entertainment, and Media Academy (AEM at John Muir High School) 

• Business and Entrepreneurship Academy (BE at John Muir High School) 

• Creative Arts, Media, and Design Academy (CAMAD at Pasadena High School) 

• Engineering and Environmental Science Academy (EESA at John Muir High 
School) 


EFFECT OF LINKED LEARNING 
CERTIFIED PATHWAYS 


A-2 


Porterville Unified School District 

• Academy of Engineering (Harmony Magnet Academy) 

• Academy of Performing Arts (Harmony Magnet Academy) 

• Digital Design and Communication Academy (DDC at Granite Hills High School) 

• Multimedia and Technology Academy (MTA at Monache High School) 

• Partnership Academy of Business (PAB at Porterville High School) 

• Health Careers Academy (Porterville High School) 

West Contra Costa Unified School District 

• Engineering Academy (Richmond High School) 

• Law Academy (Richmond High School) 

• Multimedia Academy (Richmond High School) 
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APPENDIX B. DESCRIPTION OF LINKED 
LEARNING CERTIFIED PATHWAYS 
INCLUDED IN THE STUDY 


The following information was drawn primarily from school websites and news articles. 


ANTIOCH UNIFIED SCHOOL DISTRICT 

Dozier-Libbey Medical High School 

The Dozier-Libbey Medical High School, an autonomous, stand-alone high school, opened 
in August 2008 and currently serves 640 students in grades nine through 12. Enrollment is 
determined through a district- wide application and lottery system. The Health Science and 
Medical Technology Pathway at Dozier-Libbey Medical High School received Linked 
Learning certification in the 2010—11 AY. 

The curriculum prepares students for careers in health-related fields by offering specialized 
courses and projects exploring medical careers, ethical and legal practices, global medicine, and 
employability skills. Each year’s curriculum is structured around a particular theme: nutrition 
and fitness (grade nine), complementary and alternative medicine (grade 10), life cycle (grade 
11), and medical ethics (grade 12). In grade nine, students also take a medical terminology 
course offered by Los Medanos Community College and may receive up to three college credits 
in receipt of a B grade or higher. Throughout their four years, students take courses in advanced 
math and two additional years of science, which exceeds the University of California (UC) “a— g” 
course requirements. The recommended four-year course sequence is as follows: 

• Grade nine — English 9, Algebra I or Geometry, Biology, Foreign Language or 
Visual/Performing Arts, Physical Education, Health Science 1, Explorations, 

Guided Study Tours 

• Grade 10 — English 10 or Honors English, Geometry or Algebra II, Chemistry or 
Honors Chemistry, Foreign Language or Visual/Performing Art, World History or 
AP World History, Physical Education /Health Education, Health Science 2, 
Explorations, Guided Study Tours, Community Service /Service Learning, e— mentoring 
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• Grade 11 — English 11 or AP English, Algebra II/Trig or Pre-Calculus, Human 
Anatomy Physiology, AP Biology (elective), Regional Occupational Programs 
(ROP) (Emergency Medical Care, Sports Medicine, Veterinary Science), Foreign 
Language or Visual/Performing Arts, U.S. History or AP U.S. History, Health 
Science 3, Community Service /Service Learning, Summer Externships, Job 
Shadowing 

• Grade 12 — English 12 or AP English, Pre- Calculus /AP Calculus/ AP 

Statistics /Medical Math, Physics, AP Biology (elective), ROP (Emergency Medical 
Care, Sports Medicine, Veterinary Science), Government/ Civics Economics, 
Medical Ethics, Integrated Externships, Internships, Community Service/ Service 
Learning, Volunteer Activity, Employment 

Throughout the four-year program, students engage in various work-based learning 
activities, such as guided study tours, guest speakers, internships, job shadowing, and 
community service learning. The biannual Career, College, and Community (C3) Day is an 
opportunity for students to hear, speak with, and learn from college representatives, college 
students, and health professionals. The school also strongly encourages student membership 
in the Health Occupations Students of America (HOSA). 

Dozier-Libbey Medical High School partners with many local industry businesses and 
organizations, including Los Medanos College; Facing History and Ourselves; Kaiser 
Permanente, Antioch; Emerald HPC International, Inc.; California State University (CSU), 
East Bay; Sutter Delta Medical Center; Labor Occupational Health Program, UC Berkeley; 
Community Clinic Consortium/ California Area Health Education Center Program; John 
Muir Health; Health Occupation Students of America (HOSA): Future Health Professionals; 
Costa Medical Career College; and Contra Costa County Office of Education. 
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LONG BEACH UNIFIED SCHOOL DISTRICT 

Architecture, Construction, and Engineering Academy (ACE) at 
Jordan High School 

Established in 2007, the Architecture, Construction, and Engineering Academy (ACE) 
operates as a school- within-a-school at Jordan High School. The ACE Academy serves 
approximately 280 students in grades nine through 12. A California Partnership Academy, 
the ACE Academy received Linked Learning certification in 2010. Prospective students must 
apply to ACE as a secondary specialized program through Long Beach Unified School 
District’s School of Choice application, complete an additional supplemental application, 
and hold at least a 2.0 GPA. 

The curriculum at ACE Academy meets the UC “a— g” course sequence requirements while 
also preparing students for a variety of post-graduation pathways, including college, trade 
and technical certification programs, apprenticeships, and entry-level jobs. All courses 
offered at the ACE Academy, including the academic core, are available only to students 
enrolled in the academy. Every year, students in each grade must work collaboratively to 
complete an interdisciplinary, environmentally-based project, such as the grade-nine 
“WIND” project turbine. Students in grades 11 and 12 have the opportunity to dual enroll at 
CSU Long Beach and Long Beach Community College. 

ACE Academy’s sequence of courses is as follows: 

• Grade nine — English 1—2; Biology 1—2; Reading or Literature Workshop; Core 
Career and Technical Education (CTE) Wheel: Architecture, Construction, or 
Engineering (one semester); ACE Algebra 1—2 or ACE Geometry 1—2; Health (one 
semester); Physical Education 

• Grade 10 — English 3—4; Environmental Science; Modern World History; Core 
CTE: Construction 1—2; ACE Major: Architecture 1—2, Construction 3—4, or 
Engineering 1—2; ACE Geometry 1—2 or ACE Algebra III; Foreign Language; 
Physical Education or Athletics 

• Grade 11 — English 5-6; ACE Chemistry; U.S. History; Core CTE: Architecture 
1—2; ACE Major: Architecture 3—4, Construction 5—6, or Engineering 3—4; ACE 
Algebra II or ACE Pre— Calculus; Foreign Language 

• Grade 12 — Rhetoric and Composition; ACE Government and Economics; ACE 
Physics; Core CTE: Engineering 1—2; ACE Major: Architecture 5—6, Construction 
7—8, or Engineering 5—6; ACE Pre— Calculus or ACE Calculus; Foreign Language 
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Among the work-based learning opportunities provided to ACE students are field trips — 
past trips have been to construction sites to observe “green” building and to wetlands to 
study wildlife preservation — guest speakers, mentorships, and internships for the academy’s 
grade-11 and 12- students. ACE, in partnership with the Port of Long Beach, CSU Long 
Beach, and Long Beach Community College, also hosts a variety of lectures and workshops 
aimed at building students’ employability skills and introducing them to various career paths 
in the fields of architecture, construction, and engineering. Prior to graduation, students are 
also required to have completed at least 40 hours of community service. 

ACE partners include Turner Construction; CSULB-Division of Academic Affairs; IBEW 
Local II Los Angeles Electricians; JCA Resources, Inc.; LBCC School of Trades and 
Industrial Technologies; Pacific Gateway Workforce Investment Network; SimonGlover 
Inc.; UA Local 250 of So. Ca. Steamfitters /AC /Refrigeration. 

California Academy of Math and Science (CAMS) 

The California Academy of Math and Science (CAMS), a magnet school located on the CSU 
Dominguez Hills campus, was established in 1990 and serves approximately 670 students in 
grades nine through 12. The Engineering and BioScience Pathway at CAMS received Linked 
Learning certification in the 2010—11 AY. Incoming grade-nine students with at least a 3.0 
GPA are eligible for CAMS enrollment. Students may apply to CAMS as a Secondary 
Specialized Program through Long Beach Unified School District’s School of Choice 
application and must also complete an additional supplemental application. Since CAMS is 
located on the CSU Dominguez Hills campus, students may dual enroll in college courses. 

The core curriculum is as follows: 

• Grade nine — Geometry, Accelerated Biology, Engineering Design, Accelerated 
English, Computer Science, Foreign Language 

• Grade 10 — Intermediate Algebra, Biotechnology 1—2, Principles of Engineering, 
Accelerated English, Accelerated Modern World History, Foreign Language 

• Grade 11 — Pre— Calculus, Honors Chemistry, Biotechnology 3—4, Honors English, 
Honors U.S. History, Foreign Language (recommended) 

• Grade 12 — AP Calculus, Physics/AP Physics /University Physics, AP English 
Literature, U.S. Government and Economics, Foreign Language (recommended) 
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Community of Musicians, Performers, Artists, and Social Scientists 
(COMPASS) at Millikan High School 

A Secondary Specialized Program at Robert A. Millikan High School, the Community of 
Musicians, Performers, Artists, and Social Scientists (COMPASS) integrates a college 
preparatory curriculum with social sciences and the arts to support students interested in 
careers in the arts, media, and entertainment. Granted Linked Learning certification in the 
2010—11 AY, COMPASS serves approximately 660 students in grades nine through 12. 
Students with at least a 2.5 GPA are eligible to apply to COMPASS as a Secondary 
Specialized Program through Long Beach Unified School District’s School of Choice 
application. 

In addition to the UC a— g course sequence, students enrolled in COMPASS take specialized 
courses related to the pathway students select in grade nine to pursue throughout the four- 
year curriculum. Students may select one of the six following pathways in visual or 
performing arts: fine arts, graphic arts, drama, dance, instrumental music, and vocal music. 
The recommended four-year course sequence for the core curriculum is as follows: 

• Grade nine — English 1—2; Algebra 1—2 or AB Geometry 1—2; Earth Lab Science; 
German 1—2, Spanish 1—2, French 1—2, or Italian 1—2; Physical Education; Health or 
Computer Applications 

• Grade 10 — Modern World History or AP World History; English 3—4; Geometry 
1—2 or Intermediate Algebra; Biology; German 3—4, Spanish 3—4, French 3—4, or 
Italian 3—4; Physical Education 

• Grade 1 1 — U.S. History or AP U.S. History; English 5-6 or AP English Language; 
Intermediate Algebra or Pre-Calculus; Chemistry; German, Spanish, French, or 
Italian (recommended) 

• Grade 12 — Government or Economics (AP options available); Rhetoric and 
Composition, Multicultural Literature, Film Analysis, or AP English Literature; 
fourth year of math (recommended); fourth year of science (recommended) 

The recommended CTE and elective courses by pathway are as follows: 

Visual Arts: Fine Arts 

• Grade nine — Drawing and Painting 1—2 

• Grade 10 — Drawing and Painting 3-4 or 3-D Art 1-2 

• Grade 11 — Drawing and Painting 5—6 or 3— D Art 1—2 or 3—4 

• Grade 12 — Drawing and Painting 7-8 or AP Art Studio 


EFFECT OF LINKED LEARNING 
CERTIFIED PATHWAYS 


B-6 


Performing Arts: Drama 

• Grade nine — Drama 1—2 

• Grade 10 — Advanced Drama 

• Grades 11 and 12 — Advanced Drama or Play Production 
Performing Arts: Dance 

• Grades nine and 10 — Beginning Dance, Intermediate Dance, or Advanced Dance 

• Grades 11 and 12 — Intermediate Dance or Advanced Dance 

Performing Arts: Vocal Music 

• Grade nine — Cecilian Singers 1—2, Varsity Chorale 1—2, or Voice/ Chorus 

• Grade 10 — Cecilian Singers 3—4, Varsity Chorale 3—4, or Concert Choir/ Vocal 
Ensemble 

• Grade 11 — Cecilian Singers 5—6, Varsity Chorale 3—4, or Concert Choir/Vocal 
Ensemble 

• Grade 12 — Cecilian Singers 7—8, Varsity Chorale 3—4, or Concert Choir/Vocal 
Ensemble 

Performing Arts: Instrumental Music 

• All four years — Jazz Ensemble, Symphonic Winds, Concert Band, Orchestra, or 
Marching Band (one semester) 

• Ideological Capstone and Suggested Social Science Electives 

• Grade 1 1 — AP Psychology, Creative Writing, AP Music Theory, AP Art History, or 
Yearbook Production 

• Grade 12 — Sociology and Current Events (Capstone course), Creative Writing, AP 
Psychology, AP Art History, or Yearbook Production 

Community service and enrichment opportunities are also integral to the COMPASS 
programming. Opportunities include an arts partnership with Emerson Parkside Academy 
Charter School for grade-nine students, a visit to the World War I Museum in grade 10, arts 
partnerships with Mary Bethune Transitional Center for the Homeless and Comprehensive 
Child Development Center in grade 12, and volunteering as Beach Walk organizers for the 
Children’s Hospital, also in grade 12. In addition, work-based learning courses are available 
to students in grades 11 and 12 and include careers with children and graphic design. 
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Personal Success through Empowerment, Academic Achievement, 
Conflict Resolution, and Ethics in Action (PEACE) at Millikan High 
School 

A small learning community within Millikan High School, Personal Success through 
Empowerment, Academic Achievement, Conflict Resolution, and Ethics in Action 
(PEACE) serves approximately 750 students in grades nine through 12. Granted Linked 
Learning certification in the 2010—11 AY, PEACE emphasizes careers in government, 
nonprofit organizations, and legal services. Students with at least a 2.75 GPA are eligible for 
enrollment at PEACE. Students may apply to PEACE as a secondary specialized program 
through Long Beach Unified School District’s School of Choice application. 

PEACE’S specialized curriculum focuses on classes and projects designed to highlight 
current social issues, community involvement, leadership, and collaboration. While PEACE 
students take core curriculum classes (English, history, math, and science) with their PEACE 
classmates, they take their elective classes (PE, visual/performing arts, foreign language) with 
the entire high school cohort. Students complete one major project every year that integrates 
academics with the public service pathway: Energy Project (grade nine); Heifer International 
Project (grade 10); Mock Trial Project (grade 11); ELEVATE Senior Project (grade 12). The 
standard course sequence is as follows: 

• Grade nine — English 1—2, Algebra 1—2 or Geometry 1—2, Earth Lab Science, World 
Languages 1—2 (Spanish, French, German, or Italian), Visual/Performing Arts 
elective, Physical Education or sports. In addition, grade-nine students can take 
Geography or Computer Applications as college-preparatory electives. 

• Grade 10 — Modern World History: International Negotiations or AP World 
History, English 3—4, Geometry 1—2 or Intermediate Algebra, Biology, World 
Languages 3—4, Visual/Performing Arts elective, Physical Education or sports. 
College-preparatory electives available to grade- 10 students are Sociology and Intro 
to Public Service (Career Exploration), both one-semester long. 

• Grade 11 — U.S. History: Focus on Women or AP U.S. History, English 5-6: Focus 
on Women or AP English Language, Intermediate Algebra or Pre- Calculus, 
Chemistry, World Languages 5—6. Grade 11 -students may take Criminal/ Civil Law 
and/ or Psychology as one-semester long college-preparatory electives. 

• Grade 12 — one semester of Economics: International Negotiations or AP 
Economics, one semester of Government or AP Government (U.S. or Comparative), 
ERWIC (College prep reading and writing) or AP English Literature, Pre-Calculus or 
AP Calculus, Science elective, World Languages 6—7 or AP World Language (Spanish 
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Language, Spanish Literature, French, or German). Grade- 12 students have the 
option to take Forensics as a college-preparatory elective, but all seniors must 
complete a Philosophy Capstone their senior year in order to graduate. 

Work-based learning opportunities extend throughout the four-year curriculum and include 
various service learning projects, career days, and guest speakers. Students are also expected 
to complete at least 115 hours of community service by graduation. 

PEACE’S partners include Pleifer International, Mary Bethune Transitional Center for the 
Homeless, Boeing, Long Beach Education Foundation, Long Beach Rescue Mission, and 
Miller’s Children’s Hospital. 
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OAKLAND UNIFIED SCHOOL DISTRICT 

Skyline Education Academy at Skyline High School 

Skyline Education Academy, a California Partnership Academy, is a small learning 
community at Skyline High School. Through a three-year program, serving approximately 
130 students in grades 10 through 12, the academy aims to prepare students for a variety of 
careers in education and child and family psychology. Skyline Education Academy received 
Linked Learning certification in the 2011—12 AY. Enrollment is determined through 
Oakland Unified School District’s Options Enrollment, which uses a ranking and lottery 
system to place students in district high schools. 

The academy’s core and elective classes are available exclusively to students enrolled in the 
academy. The sequence of courses is as follows: 

• Grade 10 — English 2, World History, Chemistry, Introduction to Education 

• Grade 11 — English 3, U.S. History, Educational Psychology 

• Grade 12 — English 4, Government/Economics, Peer Education 

Through Exploring College, Career and Community Options (ECCCO), the Skyline 
Education Academy provides its students with college and career preparation through career 
explorations, college readiness, and internships, which students participate in during 
grade 12. 

Skyline Education Academy partners include Teach Tomorrow in Oakland, Junior 
Achievement, One Land One People Youth Center, Oakland Kids First PASS Program, 
Merritt College Department of Child Development, Mills College Department of Education, 
Tobacco Use Prevention Education (TUPE), buildOn, and TRIO. 

Life Academy of Health and Bioscience 

Established in fall 2001, Life Academy is a small, autonomous public high school that serves 
approximately 170 students in grades nine through 12. A California Partnership Academy, 
Life Academy received Linked Learning certification in January 2011 for its Life Academy of 
Health and Biosciences Pathway. Student enrollment at Life Academy is determined through 
the Oakland Unified School District’s (OUSD) Options Enrollment. In the 2013-14 AY, 

Life Academy additionally served grades six and seven, and expansion to additional middle 
school grades will continue in future years. 

Graduation requirements at Life Academy are aligned to UC and CSU entrance requirements. 
Students at Life Academy select one of three career pathways to specialize in: medicine, 
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biotechnology, or mental health. In grades 11 and 12, students are required to complete 
internships related to their chosen pathway and present on their internships at the Spring 
Health Fair and Internship Expo. In order to complete each academic course, students must 
pass assessments called “certifications,” which can be in the form of tests, projects, or 
presentations. Students must also successfully pass two major performance reviews called 
“defenses” in grade 10 and grade 12. The grade 10 defense is an interdisciplinary presentation 
centered on a topic in mental illness, and the grade 12 defense, the Senior Investigative 
Project, is a research project and oral presentation on a topic related to the students’ 
internships. Additionally, students are required to take two science courses in grades 11 and 
12. A typical four-year course sequence is as follows: 

• Grade nine — English, Social Science, Algebra or Geometry, Biology, Physical 
Education or Art, Ethnic Studies 

• Grade 10 — Humanities, Geometry or Algebra II, Advanced Biology, Physical 
Education or Art, Spanish I 

• Grade 11 — Humanities, Algebra II or Pre-Calculus, Chemistry, Physiology I, 
Internship, Spanish II 

• Grade 12 — English, College Writing, Government/Economics, Physics, 

Physiology II, Internship, AP Spanish (optional) 

Life Academy’s industry partners include Oakland Children’s Hospital, Youth Bridge of Alta 
Bates, Highland Hospital, FACES for the Future, Planned Parenthood, Home Project: A 
Program of Alternatives in Action, Mercy Retirement and Care Center, Chabot Space and 
Science Center, Alternatives in Action, and the Native American Health Center. 
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PASADENA UNIFIED SCHOOL DISTRICT 

Arts, Entertainment, and Media Academy (AEM) at John Muir High 
School 

A small learning community within John Muir High School, the Arts, Entertainment, and 
Media Academy (AEM) was established in fall 2008 and serves approximately 470 students 
in grades nine through 12. Students enroll in one of the school’s three Linked Learning 
pathways. Students may apply to the AEM Academy through Pasadena Unified School 
District’s Open Enrollment program. A California Partnership Academy, the AEM Academy 
received Linked Learning certification in the 2010—11 AY. 

The AEM Academy curriculum exceeds the a— g course requirements for UC admissions. 
Specialized courses and electives center on the exploration of three relevant career pathways: 
performance, film and video production, and graphic design and visual fine art. The standard 
course sequence is as follows: 

• Grade nine — English 1 /Puente English 1, Algebra 1 / Geometry, Biology, 
Introduction to Art and Media Design, Physical Education 

o Support Courses/Additional Electives: Algebra Applications, Music 

Appreciation, Journalism News, Achievement via Individual Determination 
program (AVID) 9, English 1 Extension 

• Grade 10 — English 2/Puente English 2, Algebra 1 /Geometry/ Algebra 2, 

Biology/ Chemistry, World History/ AP European History, Film and Video 
Production, Physical Education, World Language (Spanish 1—3/Spanish Native 
Speaker 2— 3/AP Spanish), Visual/Performing Art (Chorus, Advanced Chorus, 
Beginning Band, Advanced Band, Music Appreciation) 

o Support Courses /Additional Electives: Composition II, AVID 10, Animation, 
Graphic Design, Government Fund (ASB), Graphic Design 2 (Yearbook), 
Journalism News 

• Grade 11 — English 3/AP English Language, Geometry/ Algebra 2/ Advanced 
Math/AP Calculus, Chemis try /Physics /Bio tech/ AP Biology/AP Chemistry, US 
History/AP US History, Film and Video Production, World Language (Spanish 
1-3/Spanish Native Speaker 2-3/AP Spanish), Visual/Performing Art (Chorus, 
Advanced Chorus, Beginning Band, Advanced Band, Music Appreciation) 
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o Support Courses /Additional Electives: Composition III, Studio Art, Ceramics, 
AP Studio Art, AVID 11, Animation, Graphic Design, Government Fund 
(ASB), Graphic Design 2 (Yearbook), Journalism News 

• Grade 12 — English 4/AP English Language, Geometry/ Algebra 2/ Advanced 
Math/AP Calculus, Chemistry/Physics /Bio tech/ AP Biology/AP Chemistry, 
Government, Film and Video Production 2, World Language (Spanish 1— 3/Spanish 
Native Speaker 2— 3/AP Spanish), Visual/Performing Art (Chorus, Advanced 
Chorus, Beginning Band, Advanced Band, Music Appreciation) 

o Support Courses/Additional Electives: Composition III, Studio Art, Ceramics, 
AP Studio Art, AVID 12, Animation, Graphic Design, Government Fund 
(ASB), Graphic Design 2 (Yearbook), Journalism News 

Beginning in grade 10, students begin to build their portfolios, which include a resume and 
samples of course-based projects. Students are also expected to complete 100 hours of 
community service by the end of grade 10. In grade 11, students begin an internship program, 
which extends through the end of grade 12. In addition to completing both their internship 
and portfolio, grade- 12 students must complete a senior project. They are also encouraged to 
take courses at Pasadena City College. Upon successful completion of the AEM curriculum, 
students receive both an AEM medallion and a specialized high school diploma. 

Guest speakers, field trips, job shadows, and technical guidance are offered to students each 
year. The academy also offers the ECCCO curriculum, which provides students with a 
Career Development and a Career Exploration Visit to a local business each year in grades 
10 through 12, as well as a college visit in grades 10 and 11 and an internship in grade 12. 

CTE Career Essentials are offered to students all four years. Students also take a specialized 
work-based learning course every year: Career Awareness (grade nine); Career Exploration 
(grade 10); Career Preparation: Practicum and Internships (grade 11); and Career Training 
(grade 12). 

Business and Entrepreneurship Academy (BE) at John Muir High 
School 

Established in fall 2008 as a small learning community at John Muir High School, the 
Business and Entrepreneurship Academy (BE) serves approximately 350 students in grades 
nine through 12. Students apply to BE through Pasadena Unified School District’s Open 
Enrollment program. All students attending John Muir High School are enrolled in one of 
the three Linked Learning pathways. A California Partnership Academy, the Business and 
Entrepreneurship Academy received Linked Learning certification in the 2010—11 AY. 
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The curriculum at the Business and Entrepreneurship Academy is modeled after the National 
Academy Foundation (NAF) curriculum, providing its students with coursework aligned to the 
UC “a— g” course sequence requirements and additional specialized courses designed to develop 
students’ business and entrepreneurial skills. Specialized CTE courses cover the following 
topics: Accounting; Business Communications; Business Management; Finance; Financial 
Literacy; Introduction to Business; Marketing; Small Business Ownership; and Stocks, Bonds, 
and Investments. The standard course sequence is as follows: 

• Grade nine — English 1 /Puente English 1, Algebra 1 / Geometry, Biology, Principles 
of Finance, Physical Education 

o Support Courses/Additional Electives: Algebra Applications, Spanish 1, 

Spanish Native Speaker 2, AVID 9, English 1 Extension 

• Grade 10 — English 2/Puente English 2, Algebra 1 / Geometry /Algebra 2, 

Biology/ Chemistry, World History/ AP European History, Principles of Finance, 
Physical Education, World Language (Spanish 1— 3/Spanish Native Speaker 2— 3/AP 
Spanish), Visual/Performing Art (Chorus, Advanced Chorus, Beginning Band, 
Advanced Band, Music Appreciation) 

o Support Courses /Additional Electives: Composition II, AVID 10, Animation, 
Graphic Design, Government Fund (ASB), Graphic Design 2 (Yearbook), 
Journalism News 

• Grade 11 — English 3/AP English Language, Geometry/ Algebra 2/ Advanced 
Math/AP Calculus, Chemistry/Physics/Biotech/AP Biology/AP Chemistry, US 
History/AP US History, Principles of Finance, World Language (Spanish 1—3/ 

Spanish Native Speaker 2— 3/AP Spanish), Visual/Performing Art (Chorus, 

Advanced Chorus, Beginning Band, Advanced Band, Music Appreciation) 

o Support Courses /Additional Electives: Composition III, Studio Art, Ceramics, 

AP Studio Art, AVID 11, Animation, Graphic Design, Government Fund 
(ASB), Graphic Design 2 (Yearbook), Journalism News 

• Grade 12 — English 4/AP English Language, Geometry/ Algebra 2/Advanced 
Math/AP Calculus, Chemistry/Physics/Biotech/AP Biology/AP Chemistry, 
Economics, Business Management/Entrepreneurship, World Language (Spanish 
1-3/Spanish Native Speaker 2-3/AP Spanish), Visual/Performing Art (Chorus, 
Advanced Chorus, Beginning Band, Advanced Band, Music Appreciation) 

o Support Courses/Additional Electives: Composition III, Studio Art, Ceramics, 

AP Studio Art, AVID 12, Animation, Graphic Design, Government Fund 
(ASB), Graphic Design 2 (Yearbook), Journalism News 
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Students progress through the four— year program as a cohort and complete specialized 
projects, which include the development and presentation of small business plans, creation 
and marketing of novel products, and opportunities in job shadowing, mentorships, and 
internships. In addition, students have the opportunity to work on Muir Ranch, an on- 
campus urban farm that teaches students business skills through paid internships and 
community service hours. The academy also offers the ECCCO curriculum, which provides 
students with a Career Development and a Career Exploration Visit to a local business each 
year for grades 10 through 12, as well as a college visit in grades 10 and 11 and an internship 
in grade 12. CTE Career Essentials are offered to students all four years of the curriculum. 
Students also take a specialized work-based learning course every year: Career Awareness 
(grade nine), Career Exploration (Grade 10), Career Preparation: Practicum and Internships 
(Grade 11), and Career Training (Grade 12). Upon successful completion of the BE 
curriculum, students receive both a BE medallion and a specialized high school diploma. 

Creative Arts , Media and Design Pathway (CAMAD) at Pasadena 
High School 

The Creative Arts, Media and Design Pathway (CAMAD) at Pasadena High School serves 
approximately 330 students in grades nine through 12. CAMAD received Linked Learning 
certification in the 2010—11 AY. Students apply to CAMAD through Pasadena Unified 
School District’s Open Enrollment program. 

In Grade 10, students select one of two concentrations: Visual Arts (CTE VADA) and 
Design or Graphic Communications (CTE GCA). Special opportunities available to 
CAMAD students include the opportunity to work in the student-run print shop, create 
artwork for public display, and become familiar with Adobe Creative Suite software. 
Scholarship and internship opportunities with pathway partners and local organization are 
also available. The standard course sequence is as follows: 

• Grade nine — English 1 /English 1 Honors, Algebra 1 / Geometry, Biology/AP 
Biology, Computer Applications, Physical Education, Spanish 1 /French 1 

o Support Services: Sycamores, Pasadena LEARNs, Safe School Teams, Learning 
Resource Lab 

• Grade 10 — English 2/English 2 Honors, Geometry/ Algebra 2/ Career Math, 
Biology/ Chemistry, World History/World History Honors/AP European History, 
Studio Art 1 [CTE VADA]/ Graphic Design 1 [CTE GCA], Spanish 2/French 2 

o Support Services: Sycamores, Pasadena LEARNs, SST’s, Learning Resource 
Lab, Teacher Mentoring, Online classes 
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• Grade 11 — English 3/English 3 Honors/AP English Language, Algebra 2/ 
Advanced Math, Chemistry/Physics, US History/US History Honors/AP US 
History, Studio Art 2 [CTE VADA]/ Graphic Design 2 [CTE GCA], 

Spanish 3 /French 3 

o Support Services: Sycamores, Pasadena LEARNs, SST’s, Learning Resource 
Lab, Teacher Mentoring, Online classes 

• Grade 12 — English 4/English 4 Honors/AP English Literature, Advanced Math/ 
AP Calculus, Physics/AP Physics/AP Biology, American Government/American 
Government Honors/AP Government/Economics/ Economics Honors, Portfolio 
Development [CTE VADA] /Printmaking Occupations [CTE GCA], Physical 
Education, AP Spanish, AP Art 

0 Support Services: Sycamores, Pasadena LEARNs, SST’s, Learning Resource 
Lab, Teacher Mentoring, Online classes 

The academy also offers the ECCCO curriculum, which provides students with a Career 
Development and a Career Exploration Visit to a local business each year for grades 10 
through 12, as well as a college visit in grades 10 and 11 and an internship in grade 12. CTE 
Career Essentials are offered to students all four years of the curriculum. Students also take a 
specialized work-based learning course every year: Career Awareness (grade nine); Career 
Exploration (grade 10); Career Preparation: Practicum and Internships (grade 11); and 
Career Training (grade 12). 

Engineering and Environmental Science Academy (EESA) at John 
Muir High School 

Established in fall 2008, the Engineering and Environmental Science Academy (ESSA) at 
John Muir High School serves approximately 350 students in grades nine through 12 and 
received Linked Learning certification in the 2012—13 AY. Students apply to EESA through 
Pasadena Unified School District’s Open Enrollment program. All students at John Muir 
High School are enrolled in one of the three Linked Learning pathways offered at the 
school. 

EESA is modeled after the National Academy Foundation’s Engineering Academy, which 
adopts the engineering curriculum developed by Project Lead the Way (PLTW). Specialized 
CTE courses available are Building and Construction, Environmental Science 1 (dual 
enrollment with Pasadena City College), Introduction to Engineering Design, and Principles 
of Engineering. The standard course sequence is as follows: 

• Grade nine — English 1, World Geography/ Composition, Physics, Algebra 

1 /Geometry, Biology, Introduction to Engineering Design, Physical Education 
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o Support Courses /Additional Electives: Algebra 1 Extension, Seminar 9 
(Introduction to Engineering Design Extension) 

• Grade 10 — English 2, World History, Biology, Algebra 2/ Geometry, Biology, 
Principles of Engineering, Physical Education 

o Support Courses/Additional Electives: Studio Art, Music, Spanish, 

AVID / Composition 

• Grade 11 — English 3/AP Language, US History/ AP US History, 
Chemistry/Biotech, Algebra 2/Advanced Math, Digital Electronics, Physical 
Education 

o Support Courses/Additional Electives: Studio Art, Music, Spanish, 

Construction, Auto Tech, PLTW Concentration Course, Civil and Architectural 
Engineering, Aerospace Engineering 

• Grade 12 — English 4/AP Literature, Government/Economics, AP Biology/AP 
Chemistry/Environmental Science 1, Advanced Math/AP Calculus, Engineering 
Design and Development, Physical Education 

o Support Courses/Additional Electives: AP Studio Art, Music, AP Spanish, 
Construction, Auto Tech, PLTW Concentration Course, Digital Electronics, 
Civil and Architectural Engineering, Aerospace Engineering 

Students progress through the four-year program as a cohort and complete special projects, 
receive technical mentoring, and participate in field trips, job shadowing, and internship 
opportunities. Upon successful completion of the EESA curriculum, students receive both 
an EESA medallion and a specialized high school diploma. 
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PORTERVILLE UNIFIED SCHOOL DISTRICT 

Academy of Engineering at Harmony Magnet Academy 

Harmony Magnet Academy opened in 2008 and serves approximately 500 students in grades 
nine through 12. Enrollment at Harmony Magnet Academy is determined through 
Porterville Unified School District’s Pathway Programs application, which relies on a ranking 
and lottery system to place students in district pathways. Students at Harmony Magnet 
Academy choose one of two pathways in which to enroll: the Academy of Engineering or 
the Academy of Performing Arts. The Academy of Engineering received Linked Learning 
certification in the 2010—11 AY and serves approximately 280 students. Supported by NAF, 
the Academy of Engineering offers a STEM-rich curriculum developed by Project Lead the 
Way. The typical course sequence is as follows. College preparatory courses are designated 
by a “P”: 

• Grade nine — English IP Composition or Accelerated; Integrated Math IP or 
Geometry P; Earth Science P or Chemistry P/AP; Spanish IP, Chinese IP, or 
French IP; Physical Education 1 or Health; Introduction to Engineering Design P 

• Grade 10 — English 2P Composition or Honors; Algebra 2P or Geometry P; 

Biology P/AP; World History P or Honors; Spanish 2P, Chinese 2P, or French 2P; 
Physical Education 2; Digital Electronics P or Civil Engineering and Architecture P 

• Grade 11 — English 3P Composition or AP English; Algebra 2P or Pre-Calculus P; 
Chemistry P/AP, Physiology P, or Physics P/AP; U.S. History P/AP; Spanish 3P, 
Chinese 3P, or French 3P; Physical Education 2; Aerospace Engineering P or 
Principles of Engineering P 

• Grade 12 — Expository Reading and Writing P, English 4P Composition, or AP 
English; Pre-Calculus P, Calculus P/AP, Statistics P/AP, Chemistry P/AP, Physics 
P/AP, or Biology P/AP; Spanish 4P or French 4P; Physical Education 2; Civics 
P/AP or Economics P; Bio technical Engineering P or Computer Integrated 
Manufacturing P 
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Academy of Performing Arts at Harmony Magnet Academy 

Harmony Magnet Academy, located in the Porterville Unified School District, opened in 
2008 and serves approximately 500 students in grades nine through 12. Enrollment at 
Harmony Magnet Academy is determined through Porterville Unified School District’s 
Pathway Programs application, which relies on a ranking and lottery system to place students 
in district pathways. Students at Harmony Magnet Academy choose one of two pathways in 
which to enroll: the Academy of Engineering or the Academy of Performing Arts. The 
Academy of Performing Arts received Linked Learning certification in the 2011—12 AY and 
serves approximately 200 students. The typical course sequence is as follows: 

• Grade nine — English IP Composition or Accelerated; Integrated Math IP or 
Geometry P; Earth Science P or Chemistry P/AP; Spanish IP, Chinese IP, or 
French IP; Physical Education 1 or Health; Orchestra, Dance Elements, 
Fundamentals of Guitar P, Keyboarding, Band, or Choir; Introduction to Stagecraft 
P (required Academy course); Graphic Design Production Principles P (required 
Academy course) 

• Grade 10 — English 2P Composition or Honors; Algebra 2P or Geometry P; 

Biology P/AP; World History P or Honors; Spanish 2P, Chinese 2P, or French 2P; 
Physical Education 2; Introduction to Video Production, Intermediate or Advanced 
Orchestra P, Musical Theatre Production P, Theatre Technology P, Fundamentals 
of Guitar II P, Keyboarding Skills II, Band, or Choir; Dance Elements and 
Interpretation P (required Academy course) 

• Grade 11 — English 3P Composition or AP English; Algebra 2P or Pre-Calculus P; 
Chemistry P/AP, Physiology P, or Physics P/AP; U.S. History P/AP; Spanish 3P, 
Chinese 3P, or French 3P; Physical Education 2; Motion Graphics, Advanced 
Orchestra P, Dance Technologies II P, Musical Theatre Production II P, Technical 
Theatre II, Fundamentals of Guitar II P, Keyboarding Skills II, Band, or Choir; 
Music Theory P (required Academy course) 

• Grade 12 — Expository Reading and Writing P, English 4P Composition, or AP 
English; Pre-Calculus P, Calculus P/AP, Statistics P/AP, Chemistry P/AP, Physics 
P/AP, or Biology P/AP; Spanish 4P or French 4P; Physical Education 2; Civics 
P/AP or Economics P; Advanced Motion Graphics, Advanced Orchestra P, Dance 
Technologies III P, Musical Theatre Production III, Fundamentals of Guitar II P, 
Keyboarding Skills II, Band, or Choir; Music Theory P (required Academy course); 
Performing Arts Capstone P (required Academy course) 
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Digital Design and Communication Academy (DDC) at Granite Hills 
High School 

The Digital Design and Communication Academy (DDC) at Granite Hills High School was 
established in 2009 and serves approximately 100 students in grades nine through 12. DDC 
received Linked Learning certification in the 2012—13 AY. Enrollment at DDC is 
determined through Porterville Unified School District’s Pathway Programs application, 
which relies on a ranking and lottery system to place students in district pathways. 

With an emphasis on communication, DDC offers a specialized curriculum that explores 
journalism, video production, and graphic design. Several of the elective courses offered at 
DDC are courses developed by NAF. The sequence of courses is as follows: 

• Grade nine 

• Core classes — English IP Composition or Accelerated; Integrated Math IP or 
Geometry P; Integrated Science IP or Chemistry P; Spanish IP, Spanish 
Literature IP, or French IP; Physical Education 1; Band, Choir or Engineering 
Design 1 

• Electives (all required Academy courses) — Freshman Seminar; Principles of 
Information Technology (NAF course); Digital Video; Graphic Design (NAF 
course) 

• Grade 10 

• Core classes — English 2P Composition or Honors; Algebra 2P or Geometry P; 
Biology P; World History P or Honors; Spanish 2P, Spanish Literature 2P, 
French 2P; Physical Education 2; Band, Choir, Art 2P, Computer Graphic 
Animation, Architectural Design IP, or Engineering Design 1/2 

• Electives — Video Production (NAF course); Web Design (required Academy 
course, NAF course); Beginning Journalism (required Academy course) 

• Grade 11 

• Core classes — English 3P Composition or AP; Algebra 2P, Pre— Calculus P, 
Introduction to Probability and Statistics P; Chemistry P, Earth Science P, 
Physiology P, Biology P/AP; U.S. History P/AP; Spanish 3P, Spanish Literature 
and Culture 3P/AP, French 3P; Physical Education 2; Band, Choir, Art 2P/3P, 
Architectural Design IP/2, Engineering Design 1/2, Photography P, Art 
History P/AP, or 3-Dimensional Design P 




Electives — Advanced Journalism, Advanced Video Production, Foundations in 
Visual Arts and Design P, Web Page Design P, Video Production 
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• Grade 12 

• Core classes — Expository Reading and Writing P, English 4P Composition or 
P/AP, Humanities P; Pre-Calculus P, Calculus P/AP, Introduction to 
Probability and Statistics P; Environmental Science P/AP, Physics P/AP, 
Physiology P; Spanish 4P, Spanish Literature and Culture 4P/AP, or French 4P; 
Physical Education 2, Civics P/AP, Economics P, or Psychology P/AP; Band, 
Choir, Architectural Design 2, Engineering Design 2, Photography P, Art 
2P/3P, Art History P/AP, or 3-Dimensional Design P 

• Electives — Advanced Journalism, Advanced Video Production, Foundations in 
Visual Arts and Design P, Web Page Design P 

Multimedia and Technology Academy (MTA) at Monache High 
School 

The Multimedia and Technology Academy (MTA) at Monache High School is a California 
Partnership Academy serving nearly 200 students in grades nine through 12. MTA received 
Linked Learning certification in the 2011—12 AY. The Multimedia and Technology Academy 
is also supported by NAF. Enrollment at MTA is determined through Porterville Unified 
School District’s Pathway Programs application, which relies on a ranking and lottery system 
to place students in district pathways. Work-based learning opportunities at MTA include 
field trips, guest speakers, college and worksite visits, internships, and mentorships. Students 
are also given the opportunity to participate in an on-site video production stage, website 
design, marketing opportunities with local businesses, and working with the school-wide 
news broadcast, MTV. The standard course sequence at MTA is as follows: 

• Grade nine — MTA English IP Composition or Accelerated; Algebra IP or 
Geometry P; MTA Integrated Science; Spanish IP, Spanish Literature IP, or French 
IP; Principles of Information Technology (one semester); Pathway Studies (one 
semester) 

• Grade 10 — MTA English 2P Composition or Honors; Algebra 2P or Geometry P; 
MTA Biology P/Honors/AP; MTA World History P/Honors; Spanish 2P, Spanish 
Literature 2P, or French 2P; Foundations in Visual Art and Design or Computer 
Graphics; Computer Systems (one semester); Computer Graphics (one semester) 

• Grade 11 — MTA English 3P Composition or AP English 3P Language or 
Composition; Algebra 2P or Pre-Calculus P; MTA Chemistry P/AP or Earth 
Science; MTA U.S. History P/AP; Photography P or Animation (MAYA); Video 
Production (Capstone course); Web Page Design (Capstone course); Computer 
Networking; Yearbook or Journalism 
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• Grade 12 — MTA English 4P Composition, AP English 4P Language or 
Composition, or Humanities; Pre-Calculus P, Calculus P/AP (AB or BC), or 
Introduction to Probability or Statistics; Environmental Science P/AP, Physics 
P/AP, or Physiology; Civics or Economics P/AP; Advanced Photography P or 
Advanced Animation (MAYA); Video Production (Capstone course); Web Page 
Design (Capstone course); Computer Networking 

Partnership Academy of Business (PAB) at Porterville High School 

The Partnership Academy of Business (PAB) is a small learning community at Porterville High 
School serving approximately 200 students. Enrollment at PAB is determined through 
Porterville Unified School District’s Pathway Programs application, which relies on a ranking 
and lottery system to place students in district pathways. Granted Linked Learning certification 
in the 2010—11 AY, the Partnership Academy of Business is both a California Partnership 
Academy and a NAF- supported pathway. The standard course sequence at PAB is as follows: 

• Grade nine — English IP Composition or Accelerated; Integrated Math IP or 
Geometry P; Earth Science P or Chemistry P/AP; Spanish IP or French IP; 

Physical Education 1; Band, Choir, or Art 1; Principles of Finance or Basic 
Computer Tech 

• Grade 10 — English 2P Composition or Honors; Algebra 2P or Geometry P; 

Biology P/AP; World History P or Honors; Spanish 2P or French 2P; Physical 
Education 2; Band, Choir or Art 2P; Financial Services or Financial Planning 

• Grade 11 — English 3P Composition or AP; Business Algebra P, Algebra 2P, Pre- 
Calculus P, or Statistics P/AP; Chemistry P/AP; U.S. History P/AP; Spanish 3P or 
French 3P; Physical Education 2; Band, Choir, or Art 3P; Computer Accounting 1 

• Grade 12 — Expository Reading and Writing P, English 4P Composition, or AP 
English; Pre-Calculus P, Calculus P/AP, or Statistics P/AP; Chemistry P/AP, 

Physics P/AP, Biology P/AP, Physiology P, or Earth Science P; Spanish 4P/AP 
Spanish or French 4P/AP; Physical Education 2; Civics P/AP Civics or Economics 
P; Band, Choir, or Art 3P; Applied Finance, Computer Accounting II, Retail Sales, 
Entrepreneurship, or Occupational Ethics P 

Health Careers Academy at Porterville High School 

Established in 1989, the Health Careers Academy at Porterville High School serves 
approximately 300 students in grades nine through 12 and received Linked Learning 
certification in the 2011—12 AY. The academy is both a California Partnership Academy and 
NAF-supported pathway. Enrollment at the Health Careers Academy is determined through 


EFFECT OF LINKED LEARNING 
CERTIFIED PATHWAYS 


B-22 


Porterville Unified School District’s Pathway Programs application, which relies on a ranking 
and lottery system to place students in district pathways. 

The Health Careers Academy’s curriculum is designed to allow students to become licensed 
nursing assistants and gain hospital and health career field certifications. Career exploration 
opportunities, such as mentoring, job shadowing, internships, college campus tours, and 
medical site visits, are available to students throughout their four years. In grades 11 and 12, 
students are eligible to participate in a selection of health career certification programs. 
Although these programs are available to all Porterville High School students, enrollment in 
the Health Careers Academy is preferred. In grade 1 1 , students may also enroll in the 
Nursing Assistant Program, which involves a minimum of 50 hours of classroom instruction 
and 100 hours of clinical experience over two semesters. After the program has been 
completed, students are qualified to take the national certification examination to become a 
certified nursing assistant. In grade 12, students are eligible to enroll in the emergency 
medical technician (EMT-1) training program. Upon completion of the instructional courses 
and clinical hours, students are qualified to take the national EMT certification examination. 

The standard course sequence at the Health Careers Academy is as follows: 

• Grade nine — Algebra 1 or Geometry; English IP; Careers and Integrated Health 
Sciences; Physical Education; Foreign Language; Band/Choir/Art 

• Grade 10 — Geometry or Algebra 2; English 2P; Biology; World History; Beginning 
Health Occupations; Band/Choir/Art; Ethics in the Workplace/Computers 

• Grade 11 — Algebra 2 or Pre-Calculus; English 3P; Chemistry; U.S. History; Health 
Occupations 2: Sports Medicine/Athletic Training, Certified Nursing Assistant, or 
Sports Therapy/Fitness Tech; Band/Choir/Art 

• Grade 12 — Pre-Calculus or Calculus AP/P or Statistics; English 3P; 

Civics /Economics; Band/Choir/Art; Anatomy/Physiology, Biochemistry, or 
Psychology 
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WEST CONTRA COSTA UNIFIED SCHOOL DISTRICT 

Engineering Academy at Richmond High School 

Established in fall 2011 as one of the five pathways offered at Richmond High School, the 
Engineering Academy serves approximately 100 students in grades nine through 12. The 
academy received Linked Learning certification in the 2011—12 AY. Enrollment in the 
pathway is determined in grade 10 through a ranking system. 

The curriculum at the Engineering Academy was developed by PLTW, which includes the 
following fundamental and specialized courses: 

• Fundamental Courses 

o Introduction to Engineering Design 
o Principles of Engineering 

• Specialized Courses 

o Aerospace Engineering 
o Biological Engineering 
o Civil Engineering and Architecture 
o Computer Integrated Manufacturing 
o Computer Science and Software Engineering 
o Digital Electronics 

Capstone Course — Engineering Design and Development 

Law Academy at Richmond High School 

The Law Academy, one of five pathways offered at Richmond High School, received Linked 
Learning certification in the 2010—11 AY. A California Partnership Academy, the Law 
Academy serves approximately 200 students. Enrollment in the pathway is determined in 
grade 10 through a ranking system. 

The Law Academy’s curriculum meets the UC a— g course sequence requirements as well as 
offering specialized courses and various project- and work-based opportunities. The Law 
Academy adopts the Law and Justice curriculum, developed under the Learning and 
Teaching Division at Education Development Center, Inc., which explores topics and 
careers in the legal field and judicial system. Additionally, Journey for Justice in America is a 
special CTE/ROP course offered to grade- 12 students that explores the foundations and 
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structures of the American government. CTE/ROP is the Contra Costa County Office of 
Education’s career training program designed for grades 11 and 12. 

The Law Academy course framework is as follows: 

• Grade 10 — Introduction to Law, English 2, Math, World History, Foreign 
Language, Chemistry 

• Grade 11 — Law and Justice, English 3, Math, U.S. History, Foreign Language, 
Elective/Physical Education 

• Grade 12 — Journey for Justice in America, English 4, Math /Elective, Economics, 
Elective /Physical Education, Psychology 

The three-year program provides students with field trips, guest speakers, job shadowing, 
and internship opportunities. Field trips are designed as opportunities to explore various 
legal careers and have included trips to City Hall and the Appeals Court of San Francisco, 
the Richmond Police Department and Dispatch Center, and a crime lab in Martinez. The 
program also offers students specialized hands-on activities, including crime scene 
investigations in grades nine and 10 and mock trials, which occur during all three years of the 
program. In grade 11, students also participate in Mentor Lunches, where they can engage 
with their peers and legal professionals. The Law Academy also organizes career fairs and 
summer internships for its students. 

Multimedia Academy at Richmond High School 

The Multimedia Academy is one of five pathways offered at Richmond High School that 
serves approximately 300 students in grades 10 through 12. A California Partnership 
Academy, the Multimedia Academy received Linked Learning certification in the 2010—11 
AY. Enrollment in the pathway is determined in grade 10 through a ranking system. 

The Multimedia Academy offers several CTE/ROP courses to its students that provide 
explorations into different forms of media while strengthening students’ creativity, 
communication, and storytelling skills. CTE/ROP is the Contra Costa County Office of 
Education’s career training program designed for high school juniors and seniors. The 
CTE/ROP Art and Animation course provides theoretical and hands-on training in art 
fundamentals as well as an introduction to computer graphics imaging. Studies in computer 
graphics are expanded upon in the CTE/ROP Computer Graphics Arts course. Through the 
CTE/ROP Photography/ Advanced Photography/ Advanced Digital Photography course, 
students become familiar with the fundamentals of film and digital photography. In addition 
to the CTE/ROP course offerings, the Multimedia Academy curriculum also includes the 
following electives: Multimedia 1, Multimedia 2, Photography, Theater 1, and Journalism. 
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APPENDIX C. VARIABLES USED IN ANALYSES 


Variable 

Definition 

csuag 

Indicates whether or not the student met the “a— g” course and grade point 
average (GPA) requirements for admission to the California State 

University (CSU) system, including 15 courses and a GPA of 2.0 or higher 

in these courses 

ucag 

Indicates whether or not the student met the “a— g” course and GPA 
requirements for admission to the University of California (UC), including 

15 courses and a GPA of 3.0 or higher in these courses 

csugpa 

GPA calculated for “a-g” courses required for admission to CSU 

ucgpa 

GPA calculated for “a-g” courses required for admission to UC 

hsgrad 

Indicates whether the student earned a standard high school diploma or a 

CAHSEE exempt diploma 

suspended 

Whether the student was suspended for at least one day in his or her senior 

year 

attendance 

The ratio of the number of days the student was present in school to the 
total number of days enrolled 

gender 

Indicates whether the student is male or female 

ethnicity 

Indicates the student’s race/ ethnicity. Categories were coded to Asian, 
black, Hispanic, white, and other, which included students who were 
Hawaiian/Pacific Islander, Native American or who indicated multiple 
race/ ethnicities 

prnted_lvl 

Indicates the highest level of education for the student’s parents. Parent 
education was coded as less than high school graduate, high school 
graduate, some college or an associate’s degree, college graduate, and 
graduate school or postgraduate education 

hs_ell 

Indicates whether or not the student was an English language learner (ELL) 
while in high school 

nslp_hs 

Indicates whether or not the student was eligible to participate in the 

National School Lunch Program 

everdis 

Indicates whether or not the student ever had a disability 
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Variable 

Definition 

mathtype 

Indicates whether the student’s grade-eight mathematics class was algebra I, 
geometry, algebra II or standard grade-eight mathematics 

cstmathss 

Scale score the student received on the grade-eight California Standards 

Test in mathematics 

cstengss 

Scale score the student received on the grade-eight California Standards 

Test in English 

cstmathperf 

Proficiency level achieved by the student on the grade-eight California 

Standards Test in mathematics. Categories are far below basic, below basic, 
basic, proficient, and advanced. 

cstengperf 

Proficiency level achieved by the student on the grade-eight California 

Stardards Test (CST) in English. Categories are far below basic, below 
basic, basic, proficient, and advanced. 

schlminority 

The percentage of a school’s student population that is black, Hispanic, or 

Native American based on data from the California Department of 

Education 

schl_frpm 

The percent of the school’s student population that is eligible for the 

National School Lunch Program based on data from the CDE 

tch_aveyrs 

Total years of public and/ or private educational service. Includes services in 
the current district, other districts, other states, and countries. Does not 
include substitute teaching or classified staff service. The first year of 
service is counted as one year. 

schl_locale 

Whether the school is located in an urban, suburban, town, or rural 
environment. The CCD locale variable was recoded combining codes 11, 

12, and 13 to “city;” codes 21, 22, and 23 to “suburb;” codes 31, 32, and 33 
to “town;” and codes 41, 42, and 43 to “rural.” 

districtname 

Provides the name of the school district in which the student was enrolled 
during high school 
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APPENDIX D. LOGIT REGRESSIONS USED 
TO ESTIMATE PROPENSITY SCORES 
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Table Dl: Logit estimates for predicting participation in a Linked Learning certified pathway used in propensity score matching for the academic years 2010-11, 
2011-12, and 2012-13 grade-12 cohorts 






Cohort graduation year 






AY 2010-11 



AY 2011-12 



AY 2012-13 




Standard 

Signifi- 


Standard 

Signifi- 


Standard 

Signifi- 

Variable 

Estimate 

error 

cance 

Estimate 

error 

cance 

Estimate 

error 

cance 

Gender 

Males (reference) 

- 

- 


- 

- 


- 

- 


Females 

0.620 

(0.753) 


-1.866 

-0.68 

*** 

-0.053 

(0.578) 


Race/ethnicity 

Asian (reference) 

— 

— 


— 

— 


— 

— 


Black 

-0.823 

(0.384) 

** 

-0.632 

(0.284) 

** 

-0.219 

(0.272) 


Hispanic 

0.037 

(0.307) 


0.128 

(0.217) 


0.203 

(0.215) 


White 

-0.250 

(0.339) 


-0.341 

(0.254) 


-0.407 

(0.247) 

* 

Other 

-0.756 

(0.652) 


0.514 

(0.463) 


-0.326 

(0.491) 


Parents' education 

Did not graduate from high school (reference) 

— 

— 


— 

— 


— 

— 


High school graduate 

0.155 

(0.188) 


-0.333 

(0.157) 

** 

-0.074 

(0.144) 


Some college or Associate's degree 

0.452 

(0.198) 

** 

-0.227 

(0.162) 


0.114 

(0.152) 


College graduate 

0.432 

(0.196) 

** 

-0.156 

(0.161) 


-0.025 

(0.154) 


Graduate school or postgraduate education 

0.186 

(0.217) 


-0.129 

(0.168) 


0.426 

(0.159) 

*** 

English Language Learner in high school 

-0.248 

(0.148) 

* 

-0.078 

(0.125) 


-0.043 

(0.117) 


Eligible for National School Lunch 

Program in high school 

-0.083 

(0.145) 


0.087 

(0.120) 


0.051 

(0.112) 


Ever identified as student with disability 3 

-0.197 

(0.230) 


-0.220 

(0.182) 


0.149 

(0.147) 


Mathematics test type b 

General math (grades 8 & 9; reference) 

— 

— 


— 

— 


- 

— 


Algebra 1 or higher 

0.092 

(0.867) 


3.174 

(0.626) 

*** 

2.244 

(0.616) 

*** 

See notes at end of table. 
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Table Dl: Logit estimates for predicting participation in a Linked Learning certified pathway used in propensity score matching for the academic years 2010-11, 
2011-12, and 2012-13 grade-12 cohorts— Continued 



West Contra Costa Unified - - -2.680 (0.266) *** -1.243 (0.215) 


See notes at end of table. 
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Table Dl: Logit estimates for predicting participation in a Linked Learning certified pathway used in propensity score matching for the academic years 2010-11, 
2011-12, and 2012-13 grade-12 cohorts— Continued 


Variable 



Cohort graduation year 





AY 2010-11 


AY 2011-12 



AY 2012-13 


Estimate 

Standard Signifi- 

error cance 

Estimate 

Standard 

error 

Signifi- 

cance 

Estimate 

Standard 

error 

Signifi- 

cance 

Interaction of mathematics test type 
and CST mathematics scale score b 

General math (grades 8 & 9; reference) 

— 

— 

- 

— 


- 

— 


Algebra 1 or higher and math score 

-0.001 

(0.003) 

-0.008 

(0.002) 

*** 

-0.006 

(0.002) 

*** 

Interaction between gender and race h 

Female and Asian 

-0.422 

(0.841) 

1.894 

(0.734) 

** 

0.058 

(0.633) 


Female and Black 

-0.399 

(0.815) 

2.405 

(0.725) 

*** 

0.177 

(0.627) 


Female and Hispanic 

-0.422 

(0.767) 

1.902 

(0.691) 

*** 

0.342 

(0.588) 


Female and white 

-0.471 

(0.789) 

2.470 

(0.709) 

*** 

0.541 

(0.609) 


Female and other 

— 

— 

— 

— 


— 

— 


Interaction between CST mathematics proficiency and 









CST mathematics scale score b 









Far below basic and math scale score (reference) 

— 

— 

— 

— 


— 

— 


Below Basic and math scale score 

0.024 

(0.021) 

0.021 

(0.014) 


0.016 

(0.013) 


Basic and math scale score 

0.018 

(0.020) 

0.015 

(0.013) 


0.016 

(0.012) 


Proficient and math scale score 

0.012 

(0.019) 

0.017 

(0.013) 


0.011 

(0.011) 


Advanced and math scale score c 

0.015 

(0.019) 

0.018 

(0.013) 


- 

— 



See notes at end of table. 
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Table Dl: Logit estimates for predicting participation in a Linked Learning certified pathway used in propensity score matching for the academic years 2010-11, 
2011-12, and 2012-13 grade-12 cohorts— Continued 





Cohort graduation year 





AY 2010-11 


AY 2011-12 



AY 2012-13 

Variable 

Estimate 

Standard Signifi- 

error cance 

Estimate 

Standard 

error 

Signifi- 

cance 

Estimate 

Standard Signifi- 

error cance 

Interaction between CST English proficiency 
and CST English scale score b 








Far below basic and English scale score (reference) 

— 

— 

— 

— 


— 

— 

Below Basic and English scale score 

-0.003 

(0.026) 

-0.013 

(0.018) 


- 0.000 

(0.019) 

Basic and English scale score 

-0.010 

(0.021) 

-0.007 

(0.013) 


-0.014 

(0.014) 

Proficient and English scale score 

-0.030 

(0.021) 

-0.012 

(0.013) 


-0.019 

(0.013) 

Advanced and English scale score 0 

-0.028 

(0.020) 

-0.015 

(0.012) 


— 

— 

Percent minority enrollment in school squared 

— 

— 

0.002 

(0.000) 

*** 

— 

— 

Constant 

23.587 

(14.749) 

1.239 

(4.143) 


-3.598 

(4.030) 

Number of observations 

2,391 


4,193 



3,938 



—Not available. 

a A student identified with a disability under the Individuals With Disabilities Education Act 
b Grade-eight California Standards Test (CST). 

c Proficient and advanced combined for 2013 cohort; no recoding for 2011 and 2012 cohorts. 
d Minority is defined as students who are black, Hispanic, or Native American. 

e Estimate was transformed to a natural log of the estimate for 2011 cohort; no transformation made for 2012 and 2013 cohorts. 
f Estimate was transformed to a natural log of the estimate for 2011 and 2013 cohorts; no transformation made for 2012 cohort. 
g Estimate was transformed to a natural log of the estimate for 2011 and 2012 cohorts; no transformation made for 2013 cohort. 
h Reference categories are the interaction of males with each race/ethnicity 

NOTE: Detail may not sum to 100 due to rounding. CST math and English proficiency levels are based on categorization of the grade eight CST scale scores in these subjects. Statistical significance 
determined by Student's t - test *p< 0.1, ** p< 0.05, *** p<0.01 
SOURCE: Data collected by the Linked Learning District Initiative. 
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Table El: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2010-11 grade-12 
cohort 


Variable 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>iti 

Gender 





Male (reference) 

- 

- 

- 

- 

Fema le 

0.540 

0.532 

0.016 

0.796 

Race/ethnicity 

Asian (reference) 

- 

- 

- 

- 

Black 

0.130 

0.109 

0.060 A 

0.316 

Hispa nic 

0.560 

0.570 

0.020 

0.751 

White 

0.202 

0.211 

0.023 

0.723 

Other 

0.020 

0.023 

0.020 

0.754 

Race/ethnicity by gender 

Asian female (reference) 

- 

- 

- 

- 

Black female 

0.074 

0.065 

0.036 

0.555 

Hispa nic fema le 

0.292 

0.295 

0.006 

0.926 

White fema le 

0.110 

0.111 

0.004 

0.949 

Other fema le 

0.012 

0.013 

0.012 

0.850 

Pa rents' Education 

Less than high school graduate (reference) 

- 

- 

- 

- 

High school graduate 

0.212 

0.207 

0.013 

0.842 

Some college/AA degree 

0.214 

0.196 

0.044 

0.490 

Col lege graduate 

0.272 

0.278 

0.014 

0.833 

Graduate school/postgraduate education 

0.152 

0.163 

0.030 

0.627 

English Language Learnerin high school 

0.396 

0.405 

0.019 

0.765 

Eligible for National School Lunch Program 

0.660 

0.641 

0.040 

0.539 

Ever identified as studentwith disability 6 

0.072 

0.058 

0.049 

0.371 

Type of CST Mathematics Test c 

General Math (reference) 

- 

- 

- 

- 

Algebra 1 or higher 

0.448 

0.469 

0.042 

0.507 

CST Mathematics Scale Score c 

346.740 

349.210 

0.040 

0.532 

CST English Scale Score c 

346.410 

350.310 

0.073 A 

0.215 

Interaction of CST Mathematics Test 

Type and CST Mathematics Scale Score 0 

General Math (reference) 

- 

- 

- 

- 

Algebra 1 or higher 

163.170 

170.390 

0.039 

0.544 

CST Mathematics Proficiency Level 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.188 

0.185 

0.006 

0.915 

Basic 

0.344 

0.349 

0.011 

0.867 

Proficient 

0.316 

0.321 

0.010 

0.878 

Ad va need 

0.106 

0.115 

0.031 

0.655 


See notes at end of table. 
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Table El: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2010-11 grade-12 
cohort— Continued 


Variable 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>ui 

CST English Proficiency Level 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.106 

0.089 

0.050 

0.355 

Basic 

0.384 

0.399 

0.032 

0.623 

Proficient 

0.284 

0.298 

0.032 

0.628 

Adva need 

0.162 

0.172 

0.027 

0.680 

Interaction of CST Mathematics Proficiency 

Level and CST Mathematics Scale Score 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

53.144 

52.570 

0.005 

0.935 

Basic 

112.250 

113.800 

0.010 

0.875 

Proficient 

120.160 

121.870 

0.010 

0.879 

Adva need 

50.268 

53.793 

0.026 

0.708 

Interaction of CST English Proficiency Level 
and CST English Scale Score 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

30.216 

25.261 

0.050 

0.355 

Basic 

125.780 

130.580 

0.031 

0.636 

Proficient 

105.280 

110.640 

0.033 

0.616 

Adva need 

69.036 

73.125 

0.026 

0.685 

Log percent mi nority enrol 1 me nt i n school d 

4.237 

4.218 

0.111 A 

0.076 

Log percent school eligible for 

National School Lunch Program 

-0.433 

-0.450 

0.094 A 

0.102 

Log average number of yea rs of 
teacher experience in school 

2.644 

2.651 

0.039 

0.516 

Log teacher experience squared 

7.021 

7.055 

0.038 

0.528 

Log school enrollment 

7.902 

7.894 

0.011 

0.859 

Locale of school 

Other 

Urba n 

0.998 

0.998 

0.000 

1.000 


—Not available. 

A Absolute effect size is between .05 and .25 (also shown in bold). 

a The absolute effect size for continuous variables is equal to the absolute value of the difference in the means between 
the LLCP students and the matched comparison group students, divided by the pooled standard deviation. For categorical 
variables, the absolute effect size is the difference in proportions divided by the pooled standard deviation. 
b A student identified with a disability under the Individuals With Disabilities Education Act 
c CST is the grade-eight California Standards Test. 

d Minority is defined as students who are black, Hispanic, or Native American. 

NOTE Reference categories are the interaction of males with each race/ethnicity. CST math and English proficiency levels 
are based on categorization of the grade-eight CST scale scores in these subjects. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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Table E2: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2011-12 grade-12 
cohort 


Variable 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>iti 

Gender 





Male (reference) 

- 

- 

- 

- 

Fema le 

0.52949 

0.55145 

0.044 

0.39 

Race/ethnicity 

Asian (reference) 

- 

- 

- 

- 

Black 

0.13499 

0.13724 

0.006 

0.898 

Hispa nic 

0.56619 

0.5599 

0.013 

0.805 

White 

0.18218 

0.1848 

0.007 

0.895 

Other 

0.01966 

0.02086 

0.008 

0.867 

Race/ethnicity by gender 

Asian female (reference) 

- 

- 

- 

- 

Black fema le 

0.08257 

0.08663 

0.015 

0.776 

Hispa nic fema le 

0.28309 

0.29303 

0.023 

0.668 

White fema le 

0.11271 

0.11883 

0.020 

0.709 

Other female 

0.00524 

0.00521 

0 

0.993 

Pa rents' Education 

Less than high school graduate (reference) 

_ 

_ 

_ 

_ 

High school graduate 

0.17955 

0.16722 

0.031 

0.525 

Some col lege/AA degree 

0.20446 

0.20774 

0.008 

0.874 

College graduate 

0.25819 

0.25893 

0.002 

0.974 

Graduate school/postgraduate education 

0.19397 

0.19034 

0.010 

0.857 

English Language Learnerin high school 

0.45216 

0.44359 

0.017 

0.737 

Eligible for National School Lunch Program 

0.66972 

0.65479 

0.032 

0.538 

Ever identified as studentwith disability 6 

0.07733 

0.07063 

0.022 

0.618 

Type of CST Mathematics Test 0 

- 

- 

- 

- 

General Math (reference) 

Algebra 1 or higher 

0.61861 

0.65042 

0.066 A 

0.197 

CST Mathematics Scale Score 0 

341.19 

341.36 

0.003 

0.959 

CST English Scale Score 0 

344.94 

346.16 

0.022 

0.648 

Interaction of CST Mathematics Test Type 

and CST Mathematics Scale Score 0 


See notes at end of table. 
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Table E2: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2011-12 grade-12 
cohort— Continued 


Variable 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>iti 

General Math (reference) 

- 

- 

- 

- 

CST Mathematics Proficiency Level 

211.62 

221.66 

0.057 A 

0.258 

CST Mathematics Proficiency Level 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.19921 

0.19151 

0.019 

0.705 

Basic 

0.31193 

0.31389 

0.004 

0.934 

Proficient 

0.32372 

0.33549 

0.026 

0.625 

Adva need 

0.10092 

0.09399 

0.023 

0.649 

CST English Proficiency Level 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.1114 

0.10812 

0.010 

0.838 

Basic 

0.34862 

0.34199 

0.014 

0.785 

Proficient 

0.29227 

0.30298 

0.024 

0.647 

Adva need 

0.17562 

0.17865 

0.008 

0.877 

Interaction of CST Mathematics Proficiency 

Level and CST Mathematics Scale Score 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

55.242 

53.1 

0.019 

0.704 

Basic 

100.88 

101.48 

0.004 

0.938 

Proficient 

123.21 

127.86 

0.027 

0.613 

Adva need 

46.773 

43.583 

0.022 

0.652 

Interaction of CST English Proficiency Level 
and CST English Scale Score 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

31.582 

30.674 

0.010 

0.842 

Basic 

113.62 

111.61 

0.013 

0.8 

Proficient 

108.04 

112.01 

0.024 

0.647 

Adva need 

74.076 

75.145 

0.007 

0.897 

Percent mi nority enrol 1 me nt i n school d 

71.971 

71.041 

0.070 A 

0.201 

Percent school eligible for National School 

Lunch Program 

0.64379 

0.63708 

0.046 

0.381 


See notes at end of table. 
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Table E2: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2011-12 grade 12 
cohort— Continued 



Log of School Enrollment 7.4577 7.3797 0.096 A 0.082 

District 

Antioch Unified (reference) - - - - 

Long Beach Unified 0.45478 0.41359 0.087 A 0.105 

Oakland Unified 0.05898 0.09347 0.114 A 0.011 

Pasadena Unified 0.11796 0.10132 0.048 0.299 

Porterville Unified 0.07733 0.08829 0.038 0.438 

West Contra Costa Unified 0.15203 0.12438 0.076 A 0.118 

Percent minority enrollment squared d 5382.5 5246 0.070 A 0.206 


—Not available. 

A Absolute effect size is between .05 and .25 (also shown in bold). 

a The absolute effect size for continuous variables is equal to the absolute value of the difference in the means between 
the LLCP students and the matched comparison group students, divided by the pooled standard deviation. For categorical 
variables, the absolute effect size is the difference in proportions divided by the pooled standard deviation. 
b A student identified with a disability under the Individuals With Disabilities Education Act. 
c CST is the grade-eight California Standards Test. 

d Minority is defined as students who are black, Hispanic, or Native American. 

NOTE: Reference categories are the interaction of males with each race/ethnicity CST math and English proficiency levels 
are based on categorization of the grade-eight CST scale scores in these subjects. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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Table E3: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2012-13 grade-12 
cohort 


Variable 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>iti 

Gender 

Male (reference) 

- 

- 

- 

- 

Fema le 

0.548 

0.556 

0.016 

0.737 

Race/ethnicity 

Asian (reference) 

- 

- 

- 

- 

Black 

0.118 

0.115 

0.009 

0.833 

Hispa nic 

0.607 

0.601 

0.013 

0.787 

White 

0.160 

0.157 

0.008 

0.860 

Other 

0.019 

0.022 

0.017 

0.696 

Race/ethnicity by gender 

Asian female (reference) 

- 

- 

- 

- 

Black fema le 

0.065 

0.064 

0.001 

0.985 

Hispanic female 

0.327 

0.328 

0.002 

0.970 

White fema le 

0.094 

0.097 

0.009 

0.850 

Other fema le 

0.011 

0.011 

0.000 

0.997 

Pa rents' Education 

Less than high school graduate (reference) 

- 

- 

- 

- 

High school graduate 

0.192 

0.193 

0.004 

0.935 

Some col lege/AA degree 

0.200 

0.200 

0.000 

0.999 

Col lege graduate 

0.204 

0.210 

0.014 

0.763 

Graduate school/postgraduate education 

0.212 

0.215 

0.006 

0.900 

English Language Learnerin HS 

0.464 

0.459 

0.010 

0.829 

Eligible for free or reduced 

price lunch in high school 

0.705 

0.701 

0.009 

0.846 

Ever identified as studentwith disability 6 

0.103 

0.100 

0.011 

0.801 

Type of CST Mathematics Test 0 

- 

- 

- 

- 

General Math (reference) 

Algebra 1 or higher 

0.664 

0.674 

0.022 

0.651 

CST Mathematics Scale Score 0 

348.940 

346.190 

0.040 

0.394 

CST English Scale Score 0 

353.960 

352.640 

0.023 

0.623 

Interaction of CST Mathematics Test Type 

and CST Mathematics Scale Score 0 

General Math (reference) 

- 

- 

- 

- 

Algebra 1 or higher 

234.570 

235.090 

0.003 

0.950 


See notes at end of table. 
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Table E3: The absolute effect size of covariate means for Linked Learning certified pathway and matched 
comparison group students after balancing on the propensity score: Academic year 2012-13 grade-12 
cohort— Continued 


CST Mathematics Proficiency Level 

Linked 

Learning 

Comparison 

Group 

Absolute 

Effect Size 3 

p>iti 

CST Mathematics Proficiency Level d 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.185 

0.203 

0.043 

0.349 

Basic 

0.282 

0.273 

0.019 

0.686 

Proficient 

0.473 

0.456 

0.035 

0.467 

CST English Proficiency Level d 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

0.100 

0.113 

0.039 

0.380 

Basic 

0.327 

0.311 

0.033 

0.486 

Proficient 

0.519 

0.514 

0.010 

0.827 

Interaction of CST Mathematics Proficiency 

Level and CST Mathematics Scale Score d 

Fa r below basic (reference) 

- 

- 

- 

- 

Below basic 

51.847 

56.623 

0.041 

0.366 

Basic 

91.153 

88.275 

0.020 

0.677 

Proficient 

191.590 

185.180 

0.032 

0.511 

Interaction of CST English Proficiency Level 
and CST English Scale Score d 

Farbelow basic (reference) 

- 

- 

- 

- 

Below basic 

28.262 

31.961 

0.040 

0.374 

Basic 

106.470 

101.170 

0.035 

0.464 

Proficient 

205.630 

204.030 

0.008 

0.867 

Percent mi nority enrol 1 me nt i n school 6 

73.541 

73.085 

0.035 

0.491 

Percent school eligible for 

National School Lunch Program 

0.637 

0.630 

0.047 

0.339 

Log average number of yea rs of 

teacher experience in school 

2.572 

2.561 

0.041 

0.496 

Total school enrollment 

2156.600 

2021.700 

0.103 A 

0.045 

Distri ct 

Antioch Unified (reference) 

- 

- 

- 

- 

Long Beach Unified 

0.394 

0.393 

0.002 

0.973 

Oakland Unified 

0.085 

0.116 

0.100 A 

0.032 

Pasadena Unified 

0.216 

0.168 

0.122 A 

0.011 

Porterville Unified 

0.076 

0.076 

0.001 

0.980 

West Contra Costa Unified 

0.115 

0.128 

0.040 

0.387 


—Not available. 

A Absolute effect size is between .05 and .25 (also shown in bold). 

a The absolute effect size for continuous variables is equal to the absolute value of the difference in the means between 
the LLCP students and the matched comparison group students, divided by the pooled standard deviation. For categorical 
variables, the absolute effect size is the difference in proportions divided by the pooled standard deviation. 
b A student identified with a disability under the Individuals With Disabilities Education Act 
c CST is the grade-eight California Standards Test. 
d Proficient and advanced combined. 

e Minority is defined as students who are black, Hispanic, or Native American. 

NOTE: Reference categories are the interaction of males with each race/ethnicity. CST math and English proficiency levels 
are based on categorization of the grade-eight CST scale scores in these subjects. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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Figure FI: Histograms showing the distribution of estimated propensity scores (logits) before and after 
matching for Linked Learning certified pathway and comparison group students: Academic year 2011 
grade-12 cohort 
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Figure F2: Histograms showing the distribution of estimated propensity scores (logits) before and after 
matching for Linked Learning certified pathway and comparison group students: Academic year 2011 
grade-12 cohort 
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matching for Linked Learning certified pathway and comparison group students: Academic year 2011 
grade-12 cohort 
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APPENDIX G. DESCRIPTIVE STATISTICS 
FOR OUTCOME MEASURES FOR 
OVERALL AND MATCHED SAMPLES, BY 
COHORT 
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Table Gl: Descriptive statistics for outcome measures in the overall and matched analytic samples, by cohort 


Outcome measure 


Overall Sample 



Matched Sample 


Number 

of students 

Standard 

Mean deviation 

Percent Number 

missing of students 

Standard 

Mean deviation 

Percent 

missing 

AY 2011-12 cohort 

Attendance during academic year 3 ' b 

2,139 

95.0 

6.1 

8.1 

1,916 

95.0 

5.9 

6.5 

Ever suspended during academic year 

2,139 

5.1 

22.0 

8.1 

1,916 

5.0 

21.7 

6.5 

High school graduation status 0 

2,681 

87.5 

33.1 

1.4 

2,287 

89.4 

30.8 

0.0 

Any postsecondary education 01 ' e 

1,649 

69.9 

45.9 

19.2 

1,433 

73.4 

44.2 

15.5 

Immediate postsecondary enrollment e ' f 

1,153 

81.7 

38.7 

43.5 

1,052 

82.1 

38.3 

38.0 

Postsecondary persistence® 

1,234 

62.4 

48.5 

39.5 

1,122 

63.0 

48.3 

33.8 

AY 2012-13 cohort 

Attendance rate during academic year 3 ' h 

1,799 

94.9 

7.4 

16.2 

1,444 

94.9 

7.4 

18.0 

Ever suspended during academic year 1 

1,799 

4.8 

21.3 

16.2 

1,444 

4.3 

20.3 

18.0 

High school graduation status h 

2,565 

89.5 

30.7 

1.4 

2,187 

89.3 

31.0 

0.0 

a-g grade point average for University of California System 









(including Long Beach) 

3,341 

2.69 

0.78 

30.7 

2,669 

2.72 

0.79 

32.7 

a-g grade point average for California State University System 









(including Long Beach) 

3,341 

2.70 

0.78 

30.7 

2,669 

2.73 

0.79 

32.7 

a-g grade point average for University of California System 









(excluding Long Beach) 

2,931 

2.69 

0.78 

16.1 

2,283 

2.72 

0.79 

15.8 

a-g grade point average for California State University System 









(excluding Long Beach) 

2,931 

2.70 

0.78 

16.1 

2,283 

2.73 

0.79 

15.8 

University of California System a-g completion status 









(including Long Beach) 

3,312 

28.9 

45.3 

31.3 

2,648 

30.2 

45.9 

33.3 

California State University System a-g completion status 









(including Long Beach) 

3,312 

37.3 

48.4 

31.3 

2,648 

38.3 

48.6 

33.3 

University of California System a-g completion status 









(excluding Long Beach) 

2,902 

29.2 

45.5 

16.9 

2,262 

30.5 

46.1 

16.6 

California State University System a-g completion status 









(excluding Long Beach) 

2,902 

37.0 

48.3 

16.9 

2,262 

37.7 

48.5 

16.6 

Any postsecondary education (including Oakland Unified) 

3,178 

69.5 

46.1 

20.7 

2,816 

69.6 

46.0 

18.7 

Any postsecondary education (excluding Oakland Unified) 

2,815 

65.5 

47.5 

16.1 

2,566 

66.7 

47.1 

15.1 

Immediate postsecondary enrollment (including Oakland Unified) 

2,208 

90.4 

29.5 

44.9 

1,961 

90.4 

29.5 

43.4 

Immediate postsecondary enrollment (excluding Oakland Unified) 

1,845 

88.5 

32.0 

45.0 

1,711 

89.0 

31.4 

43.4 


See notes at end of table. 
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Table Gl: Descriptive statistics for outcome measures in the overall and matched analytic samples, by cohort— Continued 


Overall Sample Matched Sample 


Outcome measure 

Number 

of students 

Mean 

Standard 

deviation 

Percent 

missing 

Number 

of students 

Mean 

Standard 

deviation 

Percent 

missing 

AY 2012-13 cohort 









Attendance rate during academic year 3 ' h 

2,187 

94.8 

8.1 

0.0 

1,841 

95.0 

7.5 

0.0 

Ever suspended during academic year 1 

2,187 

3.9 

19.3 

0.0 

1,841 

3.7 

19.0 

0.0 

High school graduation status h 

2,616 

86.0 

34.7 

1.1 

2,266 

87.5 

33.1 

0.0 

a-g grade point average for University of California System 
(including Long Beach and Oakland) 

3,313 

2.64 

0.82 

27.9 

2,663 

2.70 

0.80 

28.8 

a-g grade point average for California State University System 
(including Long Beach and Oakland) 

3,313 

2.65 

0.82 

27.9 

2,663 

2.71 

0.80 

28.8 

a-g grade point average for University of California System 
(excluding Long Beach and Oakland) 

2,417 

2.64 

0.81 

10.4 

2,038 

2.68 

0.80 

9.1 

a-g grade point average for California State University System 
(excluding Long Beach and Oakland) 

2,417 

2.65 

0.80 

10.4 

2,038 

2.69 

0.80 

9.1 

University of California System a-g completion status 
(including Long Beach and Oakland) 

3,044 

29.0 

45.4 

33.7 

2,578 

30.5 

46.1 

31.1 

California State University System a-g completion status 
(including Long Beach and Oakland) 

3,044 

39.2 

48.8 

33.7 

2,578 

41.1 

49.2 

31.1 

University of California System a-g completion status 
(excluding Long Beach and Oakland) 

2,334 

28.2 

45.0 

13.5 

1,977 

29.5 

45.6 

11.8 

California State University System a-g completion status 
(excluding Long Beach and Oakland) 

2,334 

36.9 

48.3 

13.5 

1,977 

38.3 

48.6 

11.8 


a Ratio of days attended to days enrolled. 
b Data for Long Beach and Pasadena. 
c Data for Long Beach, Pasadena, and Porterville. 

d Any postsecondary enrollment: Identifies students who enrolled at a postsecondary institution after leaving high school based on data from the National Student Clearinghouse 
e Data for Long Beach and Porterville. 

f Immediate enrollment: Identifies students who enrolled at a postsecondary institution during the first summer or fall after leaving high school based on data from the National Student 
Clearinghouse. 

g One-year persistence: Identifies students who were continuously enrolled at a postsecondary institution for at least one year after leaving high school based on data from the National Student 
Clearinghouse for Long Beach and Porterville. 

h Includes data for Antioch, Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa. 

' Includes data for Long Beach, Oakland, Pasadena, Porterville, and West Contra Costa. 
j Includes data for Antioch, Long Beach, Pasadena, Porterville, and West Contra Costa. 

NOTE: AY means academic year. 

SOURCE: Data collected by the Linked Learning District Initiative. 
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APPENDIX H. CERTIFICATION CRITERIA 
FOR LINKED LEARNING PATHWAYS 


The Certification Criteria for Linked Learning Pathways is reprinted with permission from: 


ConnectEd: The California Center for College and Career. 2011. Certification Criteria for 
Linked Learning Pathways. Berkeley, CA: ConnectEd. 
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Criteria 

1. PATHWAY DESIGN 

The pathway is designed with a structure, governance, and program of study that provides all students 

with opportunities for both postsecondary and career success. 

1.1 Design Structure 1.1.1 Pathway theme: The pathway represents a broad theme that 

reflects high expectations for all students and teachers and appeals 
to and engages students, regardless of their prior academic achieve- 
ment. The pathway theme expands, rather than narrows, postsecon- 
dary options for all students leading to a full range of postsecondary 
and career opportunities. The theme has been thoughtfully selected 

based on students' interests as well as other criteria. 

1.1.2 Student learning outcomes: The pathway community of practice 
has established a pathway-specific set of student learning outcomes 
that describes what students should know and be able to do when 
they complete the pathway program of study. Pathway student learn- 
ing outcomes align with student outcomes articulated by the district 

(e.g., graduate profile) and by the school (e.g., ESLRs). 

1.1.3 Program of study: A 3- or 4-year industry-themed pathway serves as 
the organizational structure for a 4-year high school program of study 
based on pathway student learning outcomes and aligned with district 
graduation requirements. By design, the program of study embeds 
the UC/CSU a-g subject area requirements and ensures that all stu- 
dents, regardless of their prior academic achievement, have access to 
them. The program of study establishes conditions for linking core 
academics with technical content at each grade level. Courses are 

sequenced and coordinated. 

1.1.4 Student recruitment, selection, and placement: The pathway's 
student recruitment and selection process is formalized, ensures 
that students are aware of all their available options, and guarantees 
open access to students who make informed decisions based on their 
current interests and aspirations after high school. Through the re- 
cruitment process, counselors, advisors, and other pathway staff 
communicate that they have high expectations for all students, re- 
gardless of their prior academic achievement. Pathway demographics 
reflect those of the school and district. 

1.1.5 Cohort scheduling: Pathway students, regardless of their prior aca- 
demic achievement, participate as a cohort in the academic and tech- 
nical courses that are part of the program of study. Cohort scheduling 
enables flexible use of class time and instructional methodologies that 

promote multidisciplinary projects. 
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1.1.6 Staff collaboration: School and pathway leadership nurtures the de- 
velopment of a pathway community of practice among staff that en- 
courages ongoing teacher development as well as frequent and 
effective collaboration for program coordination, curricular integration, 
and specialized attention to individual student needs. 

1.1.7 Pathway preparation and orientation: The pathway provides an 
orientation and other transition services for incoming students and 
their families, preferably beginning in middle school. 

1.1.8 Postsecondary articulation: The pathway ensures opportunities for 
students to make a seamless transition into postsecondary education 
and training opportunities through dual enrollment, articulation 
agreements, and other formal and informal activities. 

1.2 Governance 

1 .2.1 Advisory board with broad representation: A demographically 
diverse advisory board meets regularly to set policies, develop 
resources, and provide advice on pathway student learning outcomes 
and the pathway's program of study. 


2. ENGAGED LEARNING 

In supportive learning communities, students meet technical and academic standards and college en- 
trance requirements through real-world applications, integrated project-/problem-based instruction, au- 
thentic assessments, and work-based learning. 


2.1 Standards-Aligned 

Curriculum 

2.1 .1 Academic core: The academic curriculum is rigorous, aligned to 
state and Common Core Standards, and designed to lead to student 
proficiency on standardized tests as well as on more authentic as- 
sessment measures. All pathway students — regardless of their back- 
ground, special education or English Language Learner designation/s, 
or prior academic achievement — have access to UC/CSU approved 
a-g curriculum. 

2.1.2 Technical core: The pathway includes a 3- or 4-year sequence or 
cluster of rigorous technical coursework that is aligned to state 
and/or national CTE and industry standards. All pathway students, re- 
gardless of their prior achievement, have access to all offered 

UC/CSU a-g-approved technical core curriculum in the pathway. 

2.2 College and Career 

Readiness 

2.2.1 College readiness: The pathway prepares students for success — 

without remediation — in California's community colleges, universities, 
apprenticeships, and other postsecondary programs. 
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2.2.2 

Career readiness: Technical courses deliver basic and advanced 

industry-related knowledge and transferable skills. They focus on pre- 
paring youth for high-skill, high-wage employment by using authentic 
applications that bring learning to life. 

2.3 Real-World Relevance 

2.3.1 

Real-world relevance: Academic and technical courses deliver rigor- 
ous standards-based content through authentic, career-related appli- 
cations. Pathways alter how core academic and technical subjects are 
taught, but they do not lower expectations about what is taught. 

2.4 Integrated Curriculum 

2.4.1 

Multidisciplinary integrated curriculum: Pathway students partici- 
pate in multidisciplinary projects that integrate rigorous academic and 

technical course content. 


2.4.2 

Curricular alignment: Teachers collaborate within and across discip- 
lines and grade levels to provide students with an outcomes-driven, 
coordinated, coherent, and relevant curriculum. 

2.5 Instruction and 

Assessment 

2.5.1 

Project-/Problem-based approach: Pathway teachers use inquiry- 
based instruction to engage students in authentic theme-based expe- 
riences that require them to integrate knowledge and apply skills from 
several disciplines. Research-based instructional practice is evident in 
pathway classrooms. 


2.5.2 

Authentic assessment: Pathway teachers individually and collabora- 
tively design and use a variety of formative and summative assess- 
ments to gain an accurate understanding of student learning. 
Assessments include opportunities for students to demonstrate deep 
content learning and the application of skills through authentic prod- 
ucts and performances. 

2.6 Work-Based Learning 
(WBL) 

2.6.1 

Coordinated, sequenced, and scaled: All pathway students partici- 
pate in and have access to a continuum of high-quality, real-world 
learning experiences. The sequence culminates in an extended, in- 
tensive work-related experience that may occur in a workplace, in the 
community, at school, and/or when using virtual technology. 


2.6.2 

Connected to coursework: Each WBL experience is aligned to 
pathway student learning outcomes; helps students develop transfer- 
able, applied workplace skills; and provides opportunities for them to 
apply academic and technical knowledge and skills learned in the 

classroom. 
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2.7 Personalization and 2.7.1 Culture of high expectations and support: All pathway staff indivi- 
Support Services dually hold and collectively maintain a culture of high expectations 

and support for all students. Pathway staff develop and sustain per- 
sonalized relationships with all students and foster strong connec- 
tions between students and their peers. The pathway has processes 
in place for teachers, counselors, advisors, other pathway staff, and 
partners to quickly identify and address students' academic, personal, 

and social support needs. 

2.7.2 Student motivation and engagement: Individually and collectively, 
pathway teachers consciously and consistently work to connect 
learning (both in and out of the school setting) with students' inter- 
ests in order to increase their motivation and engagement. All path- 
way students have regular opportunities to set goals, make decisions, 
and reflect on learning. 

2.7.3 Differentiated instruction: All pathway teachers routinely design 
instruction that reflects the diversity of student interests, strengths, 
and ways of learning. They design learning activities to challenge all 
pathway students and use a variety of methods to teach and assess 
learning. 

2.7.4 Academic interventions: The pathway community of practice works 
together to quickly identify students who are in need of additional 
support and/or remediation and uses a variety of timely interventions 
to ensure that all pathway students achieve pathway learning out- 
comes. These can include a variety of proactive interventions and ac- 
celeration strategies to meet individual student needs. 

2.7.5 Guidance and counseling: The pathway has a designated counselor 
and/or a system of guidance and advisement. The counselor and/or 
advisors know pathway students well and are familiar with the unique 
characteristics of the pathway program, thus enabling them to sup- 

port the success of all pathway students. 

2.7.6 College and career planning: Each pathway student has a multi-year 
college and career success plan that is informed by a range of college 
and career planning activities, extends through high school, and 
guides decisions about postsecondary education, training, and career 
pursuits. 
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3. SYSTEM SUPPORT 

District policies and practices provide leadership, support, and resources to establish and sustain quality 
pathways. 


3.1 District Policies 

3.1.1 

Pathway equity, access, and choice: District, school, and pathway 
policies and procedures support pathway development, implementa- 
tion, and sustainability. Policies and procedures allow students to 
select pathway options based on their current interests and future 
aspirations; ensure equity in the placement of students in pathways; 
and ensure that lack of transportation does not exclude students from 
participating in the pathway of their choice. 


3.1.2 

Recruitment and hiring practices: District, school, and pathway pol- 
icies and practices support the recruitment, selection, and retention 
of pathway teachers who possess the unique skills to support the 
pathway approach and reflect the demographics of the pathway, 
school, and district. District policies recognize the need for stability 
among pathway staff in order to support ongoing pathway improve- 
ment, fidelity, and sustainability. 


3.1.3 

Accountability and autonomy: The district has achieved a healthy 
balance between pathway autonomy and accountability that values 
both as necessary to improve student learning outcomes. District pol- 
icies and practices hold the pathway accountable for improving stu- 
dent outcomes and give the school and pathway the autonomy to 

determine how best to attain these outcomes. 

3.2 Leadership 

3.2.1 

Support from school board and superintendent: The district board 
of education and superintendent are strong proponents of the path- 
way approach, publicly endorse it, offer active support, and align re- 
sources and procedures to promote the quality and sustainability of 
pathways. 


3.2.2 

Support from site leadership: The high school principal and other 
administrators publicly advocate for the pathway and are actively 
involved in its funding, facilities, staffing, scheduling, partner recruit- 
ment, and other forms of support. Site leaders demonstrate a firm 
understanding of, vision for, and commitment to pathways and their 
potential to improve student learning outcomes. 

3.3 Professional 

Development 

3.3.1 

Teacher professional development: Site and district administrators 
and/or the pathway community of practice provide or coordinate on- 
going training for pathway teachers. 
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3.4 Qualified Staff 

3.4.1 Skilled teachers: A pathway's success rests on high-quality teaching 
and collaboration among an interdisciplinary team of academic and 
technical teachers. Site principals and district leaders hire, assign, and 
provide ongoing professional growth opportunities for pathway teach- 
ers who are willing, skilled, and highly qualified. 

3.4.2 Pathway leadership: District and site administrators support the se- 
lection and further development of pathway leaders who inspire and 
guide the pathway community of practice in improving pathway quali- 
ty and student success. Release time is provided for this role. 

3.5 Partnerships 

3.5.1 Active employer and community partnerships: The pathway has 
strong partnerships with local employers, community groups, and 
individuals. Both through the advisory board and other interactions, 
there is evidence of a healthy partnership between the pathway/high 
school and its host community. 


4. DATA AND IMPACT 

A systemic and systematic evaluation process documents the pathway's impact on high school 
achievement and postsecondary success and drives the pathway's continuous improvement plans. 


4.1 Student Data 

4.1 .1 Data collection: The pathway and/or district regularly collect and 
accurately report pathway students' demographic and performance 

data. 

4.1.2 Use of data: The pathway community of practice regularly analyzes 
individual student as well as disaggregated and aggregate pathway 
data and compares data on student performance across the school, 
district, and state. Pathway staff use data regularly to inform instruc- 
tional practice and curricular decisions, improve pathway student 
learning outcomes, make programmatic decisions, and support stu- 
dents' academic success. 

4.2 Pathway Evaluation 

4.2.1 Evidence of impact: Data analysis shows that the pathway retains 
participating students, demonstrates improvement in student perfor- 
mance, and eliminates opportunity and achievement gaps. 
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4.2.2 Periodic review and improvement plan: Pathway staff and the 
advisory board regularly review data and other indicators to assess 
students' progress toward achieving the pathway's learning out- 
comes. These periodic reviews result in the development of an im- 
provement plan and action items that are based on the pathway's 
data and the recommendations contained in its certification Final Re- 
port (if one already exists) and reflect the pathway's underlying mis- 

sion and goals. 

4.2.3 Postsecondary tracking: Pathway staff conduct a formal follow-up of 
students for multiple years after high school graduation and use these 
data to develop a plan for continuous improvement of the pathway 
and student learning outcomes. There is some mechanism in place to 
track a significant and representative sample of pathway students in 
order to determine their enrollment and success in postsecondary 
education and careers. 
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APPENDIX I. DEFINITIONS OF 
STATISTICAL TERMS USED 


Term 

Definition 

Average treatment effect 
on the treated (ATT) 

A treatment, or policy, effect averaged across the 
population 1 

Balance 

An indication that treatment and comparison groups are 
statistically similar across the variables used to calculate 
propensity scores 2 

Bias 

The deviation of the average value from the true population 
value. Bias refers to systematic errors that affect any sample 
taken under a specific design with the same constant error. 3 

Caliper 

A threshold or tolerance on the maximum propensity score 
distance 2 

Covariate 

An independent variable, or a variable that is used to explain 
the variation in the outcome variable 1 

Dummy variable 

Independent variables which take the value of either 0 or 1 

Just as a ’’dummy” is a stand-in for a real person, in 
quantitative analysis, a dummy variable is a numeric stand-in 
for a qualitative fact or a logical proposition. 4 

Effect size 

The standardized magnitude of the effect or the departure 
from the null hypothesis. For example, the effect size may 
be the amount of change over time, or the difference 
between two population means, divided by the appropriate 
population standard deviation. Multiple measures of effect 
size can be used (e.g., standardized differences between 
means, correlations, and proportions). 3 

Estimate 

Estimates result from the process of providing a numerical 
value for a population parameter on the basis of information 
collected from a survey and/ or other sources. 3 
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Linear model 

A statistical model where the relationship between the 
dependent variable and each independent variable is 
constant 1 

Log odds 

The probability of a binary outcome, also see logit 1 

Logistic regression 

A statistical model that predicts the outcome of a binary 
dependent variable or outcome based on linear independent 
variables 1 

Logit 

The log of the odds of an event happening to the odds of it 
not happening 1 

Marginal effect 

The effect on the dependent variable that results from 
changing an independent variable by a small amount 1 

Matching algorithm 

The process used to match treatment cases to one or more 
comparison group cases in a propensity score matching 
study 5 

Mean 

The sum all the data values divided by the number of data 
values 6 

Normalized weights 

In general, weights are relative values associated with each 
sample unit that are intended to correct for unequal 
probabilities of selection for each unit due to sample design. 
Normalized weights are adjusted such that the sum of the 
weights is equal to the number of cases in the sample . 3 

Observational data 

Data that have not been obtained through a controlled 
experiment 1 

Ordinary least squares 
(OLS) regression 

A method for estimating the parameters of a multiple linear 
regression model. The ordinary least squares estimates are 
obtained by minimizing the sum of squared residuals 1 

Propensity score 

A score that describes probability of participation in or 
receiving a specific treatment 7 


Quasi— experimental study 


An empirical study that compares treatment and comparison 
groups but lacks the critical element of random assignment 8 
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Random error 

Random error is generally different each time a 
measurement is made, and behaves like a number drawn 
with replacement from a box of numbered tickets whose 
average is zero. 9 

Randomized controlled 
trial (RCT) 

A research design in which groups are created through a 
process that is random. Carried out correctly, random 
assignment results in groups that are similar on average in 
both observable and unobservable characteristics, and any 
differences in outcomes between the groups are due to the 
intervention alone. 10 

Standard deviation 

A measure of variation across observations in a sample. A 
low standard deviation indicates that the observations in the 
sample tend to be very close to the mean. A high standard 
deviation indicates that the observations in the sample tend 
to be spread out over a large range of values. The standard 
deviation is the square root of the variance. 10 

Standard error 

The standard deviation of the sampling distribution; 3 also 
refers to the average amount of measurement error for an 
estimate 1 

Statistical adjustment 

Including baseline measures in a statistical model at the level 
of the unit of analysis. A number of different techniques can 
satisfy the statistical adjustment requirement specified by the 
What Works Clearinghouse (2014), including regression 
adjustment, analysis of covariance (ANCOVA), and 
hierarchical linear modeling. Randomized controlled trials 
with high attrition and quasi-experimental designs require 
statistical adjustment to control for differences in baseline 
characteristics if the effect size of the difference in baseline 
characteristics is between 0.05 and 0.25 standard 
deviations. 10 

Statistical significance 

The likelihood that a finding based on sample data is due to 
chance rather than a real difference in the population from 
which the sample was drawn. When the probability that a 
finding is due to random chance is less than 5 percent (or 
some other percentage), the finding is often considered to 
be statistically significant 4° 


Systematic error 


An error that affects all the measurements similarly 9 
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/test 


A statistical significance test used to test hypotheses about 
one or two means when the population standard deviation is 
unknown 1 2 3 4 5 6 7 8 9 10 11 


/ value 


A numeric criterion to determine whether the results of a 
/test are due to chance or not 11 


Variance 


The average value of the squared difference between the 
value of a variable for each member of a sample or a 
population and the average value or mean for all members 
of the sample or the population. The variance is equal to the 
square of the standard deviation . 10 
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